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ABSTRACT 


Many military Intelligence Surveillance and Reconnaissance (ISR) 
operations would benefit greatly from a fleet of disparate sensor-bearing UAVs 
that are tightly integrated via a communications network, work cooperatively for a 
common operational objective, enhance situation awareness of the areas of 
operation, and increase persistence of sensor dwell time on strategic targets. 
This would enable continuity in the entire target acquisition cycle, from detection 
to classification to identification and finally localization of targets, in a diverse and 
dynamic environment. The integration of sensors and development of tactics in a 
cooperative sensing environment is one of the current focuses among the military 
intelligence community, and hence motivates this thesis effort. By building 
models with an existing agent-based simulation platform and using an extremely 
efficient experimental design methodology, numerous factors which could 
potentially affect the effectiveness of a cooperative sensing network against two 
arrays of targets are explored. The factors considered include UAV airspeed, 
reliability, detection/classification coverage and probability, network latency and 
degradation, UAV configurations and responsiveness, as well as air space 
separation. The two arrays of targets are mobile armor concentrations and time 
critical targets; these vary in their deployment profiles, vulnerability constraints 
and ease of detectability. Factors characterizing these targets, such as the shoot- 
and-scoot behavior of time critical targets, are also investigated. The study 
provides operational insights pertaining to the design and effective use of 
cooperative sensing for ISR purposes. These include the importance of having 
good UAV sensor capabilities, the need for a suite of sensors to aid in locating 
well-camouflaged time-critical targets, and the need for “intelligent” application of 
UAV cooperation tactics based on the characteristics of recently-classified 
targets. 
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THESIS DISCLAIMER 


The reader is cautioned that computer programs developed in this 
research may not have been exercised for all cases of interest. While every effort 
has been made, within the time available, to ensure that the programs are free of 
computational and logic errors, they cannot be considered validated. Any 
application of these programs without additional verification is at the risk of the 


user. 
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EXECUTIVE SUMMARY 


As a result of combat operations during Operation Iraqi Freedom, a major 
lesson learned indicates that the success of a battle is increasingly reliant on 
more accurate and timely collection of the battle-space intelligence. Unmanned 
systems, and certainly Unmanned Aerial Vehicles (UAVs), are widely used in 
today’s military ISR operations, and will be even more prevalent in the next 
generation military as warfighting concepts such as Network Centric Warfare and 
Distributed Operation emerge. A number of significant military Intelligence 
Surveillance and Reconnaissance (ISR) operations would benefit greatly from a 
fleet of disparate sensor-bearing UAVs that are integrated via a communications 
network, work in a cooperative manner for a common operational objective, 
enhance situation awareness of the areas of operation, and increase persistence 
of sensor time on strategic targets. Furthermore, the cooperative sensing 
network enables continuity in the entire target acquisition cycle, from detection to 
classification to identification and finally localization of interested targets, in a 
diverse and dynamic environment. The integration and tactics development in a 
cooperative sensing environment are currently key focuses among the military 


intelligence community, and serve as the motivation for this thesis. 


The objective of the thesis is to explore the enhancement in detection and 
classification capabilities for mission critical and time critical targets, should the 
capability of a cooperative sensing network becomes available to the Marine 
Expeditionary Brigade commander. Two types of UAVs are considered: the Low 
Altitude and Endurance (LAE) and the Medium Altitude and Endurance (MAE). 
The two target types in the study include the time critical targets and the mobile 
armor concentrations; they represent the two main arrays of targets 
characterized by their deployment profiles, vulnerability constraints and ease of 
detectability. The study measures the proportion of the enemy’s mobile targets 
and Time Critical Targets (TCTs) classified over a four-hour period for 
Intelligence Preparation of the Battlefield prior to a Marine Expeditionary 


xvii 


amphibious assault. In doing so, the author applies suitable analytical 
methodologies to gain insights into questions as these: 


¢ What are good cooperative sensing network configurations? 
¢ Are there diminishing returns with increasing UAVs allocated? 
¢« What are good UAV responses in a cooperative sensing environment? 


« What are the effects of degraded communications and increasing 


latency? 


¢« What are the effects of UAV failure in the presence of cooperative 


sensing? 


A number of models are developed with reference to the Marine Corps 
Warfighting Laboratory's Sea Viking 2004 scenario using the agent-based 
simulation platform Map Aware Non-uniform Automata (MANA). The computing 
resource requirements for these models are much lower than for most 
conventional simulation models, and can be executed over tens of thousands 
repetitions to explore many possible representations of real-life situations Time 
and cost considerations would preclude running even a fraction of these 
situations in a field experiment, so the simulation results provide useful insights 
regarding overall system effectiveness. A total of 20 factors varying over 1290 
design points are considered. Controllable factors pertaining to the number of 
LAEs, UAV characteristics such as airspeed, detection range, classification 
range and probability of classification, network parameters including link reliability 
and message latency, as well as air-space separation are explored. Factors 
uncontrollable in real-world settings, such as enemy concealment, counter 
detection sensor range and period of vulnerability, are also varied in the 
simulation runs. The 1290 excursions with 50 repetitions each, amounting to 
64,500 MANA runs, are submitted to the MAUI High Performance Computing 
Center for computation. Another set of models and excursions are also 
generated to study the effects of an LAE failure at specific time periods during 
the four-hour operation. 


xviii 


The data collected from the simulations is analyzed using several 
graphical and statistical tools. Multiple Linear Regression and Data Partitioning 
are used in conjunction with the results from other statistical techniques to draw 
useful insights and operational guidance relevant to the employment of 


cooperative sensing for military ISR purposes. 


The results show approximately 9.8% of the TCTs and 27.2% of the armor 
targets are classified with the configuration of an MAE and four LAEs at the end 
of the four-hour operation. The relatively low classification measures are 
attributed to the relatively small coverage area extended throughout the entire 
duration of operation as compared to the considerably large area of interest. The 
analysis leads to the observation that the classification effectiveness depends on 
the UAV routings, and is an important consideration when preparing an 
intelligence collection plan. Prior knowledge of the enemy’s courses of action, 
target’s profiles and terrain information is beneficial. An LAE should be more 
responsive to the MAE cueing to achieve higher classification performance 
against TCTs and less responsive to the MAE cueing when acquiring armor 
targets. However, an LAE should tend to follow a new classified enemy contact 


when searching out armor targets as they are generally deployed in formations. 


While much is discussed about the behaviors of the UAVs in a cooperative 
sensing environment with shared situation awareness, many fundamental sensor 
characteristics like longer classification range and higher classification probability 
also have impacts on the overall classification effectiveness. However, an LAE 
classification range that is larger than 5 km easily distracts the LAE; more clutter 
appears in the sensor view and penalizes the overall classification performance. 
An increase in number of LAEs generally improves the target classification 
proportions, but the proportion of armor classification seems to taper off at four 
LAEs for this scenario. 


Link reliability and message latency are shown to have no significant 
effects on the classification outcome. Air-space separations also do not matter. 


High TCT concealment capabilities and counter-detection sensor ranges 
xix 


significantly hinder good classification performance of the sensing network. A 
UAV equipped solely with an electro-optics sensor may not be effective against 
the well-camouflage TCTs. Against such targets, other sensors or radars may be 


a more viable option. 


There is some indication showing that cooperative sensing does provide a 
more robust solution in some scenarios in terms of the classification performance 
should an LAE fail or be shot down during the four-hour operation. Following a 
fixed set of standard operating tactics regardless of the targets of interest and 
enemy’s courses of action is not the way forward in employing a cooperative 


sensing network. 
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I. INTRODUCTION 


A. OVERVIEW 


Unmanned Aerial Vehicles (UAVs) are remotely piloted or self-piloted 
aircraft that carry cameras, sensors, communication equipment or other 
payloads. They have been used primarily in Intelligence, Surveillance and 
Reconnaissance (ISR) roles since the 1950s. Since 1964 the Department of 
Defense (DoD), having realized the potential of UAVs as a key component in ISR 
missions, had 11 different types of UAV developed. However, due to acquisition 
and development problems, only three entered production. The U.S. Navy has 
also studied the feasibility of operating Vertical Takeoff and Landing (VTOL) 
UAVs since the early 1960s to overcome the limitation of short takeoff distance 
from surface vessels, the QH-50 Gyrodyne torpedo-delivery drone being an early 
example. Nonetheless, high cost and technological immaturity precluded 
acquiring and fielding operational VTOL UAV systems at that time. By the early 
1990s, DoD sought UAVs capable of satisfying surveillance requirements in 
Close Range, Short Range or Endurance categories, where persistence, 
dynamic re-tasking capability and real-time imagery are critical to the mission. By 
the late 1990s, the Close and Short Range categories were combined, and a 
separate Shipboard category emerged. The current classes of these vehicles are 
the Tactical UAV and the Long Endurance UAV, however, both classes are still 
predominately employed for ISR purposes (Federation of American Scientists 
Intelligence Resource Program website, 2005). 


The use of UAVs evolved rapidly over the last two decades to conduct air- 
to-ground strike missions with the development of the Unmanned Combat Aerial 
Vehicles (UCAVs). The operational UCAV system is envisioned as a force 
enabler for the U.S. Air Force that will conduct Suppression of Enemy Air 
Defense (SEAD) and strike missions in support of post-2010 manned strike 
packages. This SEAD/Strike mission will be the first instantiation of an UCAV 


vision that will evolve into a broader range of combat missions as the concept 
1 


and technologies mature, and the UCAV affordability potential is realized. The 
DARPA/Air Force/Boeing X-45A technology demonstration aircraft completed its 
first flight on 22 May 2002. Multi-aircraft testing began in 2003 when a second X- 
45A became operational, leading to joint UCAV and manned exercises in 2006 
(Federation of American Scientists Military Analysis Network website, 2005). As 
technology advances and the importance of unmanned vehicles as a force 
enabler is realized by the military, the Stealth UAV becomes the current class of 


UAV to be researched and developed. 


Being an unmanned platform and generally smaller in size, the UAV is 
preferred over manned aircraft for strategic deployment into the airspace deep 
behind the adversary line of operations which is often protected by heavy air 
defense elements. Today UAVs are widely deployed in Time Critical Targeting 
(TCT) operations, where long endurance and dynamic re-tasking capabilities are 
imperative to the success of acquiring of high value targets and eventually 
executing the kill chain in prosecuting these targets. The concept of cooperative 
sensing soon evolved to offer the ability to cue available and suitable joint 
sensors networked together in the theatre of war to extend the persistence of 
sensor time on target. This is currently drawing great attention of military 
strategists and technologists. Medium Altitude Endurance (MAE) UAVs have the 
capability of extended flight duration, typically 6-12 hours or longer. These are 
supplemented with the tactical Low Altitude Endurance (LAE) UAVs which 
enable near-real-time imagery and local tracking of targets. While the MAE UAVs 
have long-range deployment and wide-area surveillance or long sensor dwell 
over the target area to provide initial detections, the close-in LAE UAVs allow the 


identification, localization and tracking of interested targets with high accuracy. 


Today there are some 50 U.S. companies, academic institutions, and 
government organizations developing over 150 UAV designs. Forty of these 
companies have some 115 of these designs flying, i.e., at least one working 
prototype built. Fifteen of these companies have 26 models of UAVs in, or ready 
for, production (UAV Forum website, 2005). With recent emphasis in Network 
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Centric Warfare (NCW) which extensively exploits UAVs as an indispensable 
part of the sensor grid, the development of UAVs will definitely be an important 


part of military development and a future force multiplier. 


B. MOTIVATION 


As a result of combat operations during Operation Iraqi Freedom, a major 
lesson learned indicates that a better asset to collect battle-space intelligence is 
crucial to the way forward (U.S. Joint Forces Command, 2004). A number of 
significant military operations would benefit greatly from a fleet of disparate 
sensor-bearing UAVs that are integrated via a communications network, work in 
a cooperative manner for a common operational objective, enhance situation 
awareness of the areas of operation, and increase persistence of sensor time on 
strategic targets. Figure 1 depicts a cooperative sensing network employed in a 
Time Critical Targeting operation. Multiple UAVs allow sensing to be performed 
in parallel, thereby reducing the amount of time required to gather data. If a 
vehicle becomes disabled, the remaining vehicles can continue sensing, 
although at a reduced collection rate. Most importantly, the cooperative sensing 
network enables continuity in the entire target acquisition cycle, from detection to 
Classification to identification and finally localization of interested targets, in a 
diverse and dynamic environment. The success of cooperative sensing depends 
significantly on two key technological emphases: the development of state-of-art 
imaging sensors and the integration of sensor and ISR systems in a seamless 
and cooperative manner. The ensuing paragraphs provide further discussion on 


these technologies. 


Perhaps the need for a complementary suite of sensors is of greater 
importance for surveillance on land than for a maritime environment. The 
richness of terrain variation and coverage on the ground mean that targets 
moving across land surfaces could employ effective Concealment, Camouflage 
and Deception (CCD) tactics to prevent being detected by the adversary. The 


entire battle of ground search and detection revolves around the ability to employ 
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sensing equipments and technologies that are superior to the adversary’s CCD 
capability. Over the last two decades, the development of air-to-ground sensor 
technologies has improved by leaps and bounds. Conventional microwave radar 
is now able to detect longer and perform better in a wider range of operating 
environments. Many Electro-Optics (EO) and Infra-Red (IR) imaging sensors can 
achieve astonishing spatial resolutions of less than one meter, the Foliage 
Penetration (FOPEN) Radar and Synthetic Aperture Radar (SAR) allow detection 
of targets hidden under concealment and camouflage, Laser Radar (LADAR) 
imaging is achievable with higher spatial resolution and precision up a tenth of a 
meter, and the “through-the-wall” imaging radar is under research and 
development to provide the capability to look into buildings at standoff distances 
and build a picture of the tactical situation. The availability of a diverse range of 
air-to-ground radars and imaging sensors makes it possible for a fleet of UAVs, 
each mounted with disparate sensors, to be integrated to form a cooperative 


sensing network to collect data in a parallel, coordinated and optimal manner. 





Figure 1. A Cooperative Sensing Network Deployed for Time Critical Targeting 
Operation [Best Viewed in Color] 


The greater challenge of cooperative sensing, in the author's opinion, 
does not lie on the employment of advanced sensors as one could expect these 
sensors to be widely available in the near future given the amazing rate at which 
technology advances. The greater challenge involves seamless integration of all 
the sensors in the theatre of operations and tactics development to provide high- 
quality, uninterrupted information over the entire target life to the commander for 
making battle decisions effectively. This involves an intelligent sensing system 
that is able to efficiently task suitable and available sensors on-the-fly given 
various tactical situations. The sensing system should also be capable of 
prioritizing conflicting sensing requirements, if any should occur. The 
development of such a smart system is by no means trivial; it is not uncommon to 
have overwhelming ISR requests beyond the availability of the sensors. The tight 
integration amongst sensor systems has prompted the need for developing a 
high throughput and robust communications network. Obviously, the availability 
of such a sensing network would not be effective without restructuring the military 
ISR processes, just as adopting the Network Centric Warfare paradigm would not 
be effective without changing the traditional ways of fighting a war. 


C. BACKGROUND WORK 


There have been a number of preceding theses at the Naval Postgraduate 
School (NPS) that use an agent-based modeling approach to explore the use of 
UAVs for military intelligence gathering missions. Each thesis focuses on a 
different aspect of using UAVs on a targeted ISR operational scenario. Below are 


brief summaries of some of these works. 


Raffetto (2004) analyzes the impact various UAV capabilities have on 
intelligence gathering missions for a Marine Expeditionary Brigade (MEB) 
commander’s 2015 UAV to support rapid planning and decision making for 
multiple concurrent operations. This facilitates maneuver and_ precision 
engagement based on the UAV operations in the Sea Viking 2004 (SV04) 


scenario provided by Marine Corps Warfighting Lab (MCWL). He explores the 
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validity of current requirements and provides insights into the importance of 
various UAV characteristics, such as airspeed, endurance, sweep width, and 
sensor capability for a single UAV. The analysis seeks to measure the proportion 
of the enemy’s entities detected over time, hence the ability to gain superiority in 


situation awareness (Raffetto, 2004). 


Berner (2004) explores the effective use of multiple UAVs for the Navy’s 
Surface Search and Control mission based on a coastal environment scenario 
with dense shipping traffic and sparse enemy contacts. In addition to the impacts 
various UAV parameters have on the detection and classification effectiveness of 
the enemy contacts, he also shows the ISR effectiveness for different UAV 
tactics and combinations (Berner, 2004). However, although the contacts 
acquired by each UAV are routed to a central command, this has no influence on 
the collection plan of the other UAVs as would occur within a cooperative sensing 


network. 


McMindes (2005) looks at how UAV survivability is affected by UAV 
speed, stealth, altitude, and sensor range, as well as enemy force sensor ranges, 
probability of kill, array of forces, and numerical strength. The analysis, which is 
based upon the SV04 scenario of Raffetto (2004), concludes that a UAV is highly 
survivable with fast speed, with the exception that stealth becomes more 
important than speed alone when the enemy has extremely high-capability 
assets (McMindes, 2005). 


D. THE PROBLEM 


The Sea Viking Division of the MCWL is responsible for experiment 
planning and design of the ongoing Sea Viking live experimentation campaign. 
Sea Viking is planned and executed in two-year segments. The current focus for 
Sea Viking 2006 (SV06) is to conduct live experimentation that will assist in the 
development of operational maneuver from the sea/ship to objective maneuver 


tactics, techniques and procedures and to develop and assess experimental 
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capabilities that support these concepts. Additionally, SVO6 will examine the 
emerging Distributed Operations (DO) concept which will deploy for operations in 
the Global War on Terror (Marine Corps Warfighting Laboratory Sea Viking 
website, 2005). General Hagee, Commandant of the Marine Corps, highlights the 
required capabilities for enhanced intelligence capabilities to collect, report, and 
exploit intelligence for small units employing DO. These include employment or 
direction of unmanned ground or air vehicles, or the ability to access command 
and control networks for the purpose of extracting specific intelligence pertinent 
to the unit’s local situation (Hagee, 2005). A cooperative sensing capability will 
indeed provide the required strategic and tactical ISR requirements for the 


emerging DO concept. 


The objective of this study is to explore the enhancement in the 
effectiveness on detection and classification capabilities of mission critical and 
time critical targets, should the capability of a cooperative sensing network 
become available to the MEB commander. These targets are deemed to have 
significant implications for subsequent battle developments, and hence warrant 
elaborate sensing requirements directed toward them. The study aims to 
measure the proportion of the enemy’s mobile and time critical targets classified 
over a four-hour period for Intelligence Preparation of the Battlefield (IPB) prior to 
an amphibious assault, thus offering insights about the effectiveness of the 


cooperative sensing network for ISR missions against these target types. 


E. SCOPE AND ASSUMPTIONS 


The research focuses primarily on formulating and building models to 
capture essential characteristics of UAVs in an IPB operation. These models are 
explored extensively to uncover how the effectiveness of cooperative sensing 
capability is affected by ISR processes and structures as well as UAV responses 
to shared Situation Awareness (SA). An existing agent-based distillation called 
Map Aware Non-uniform Automata (MANA) is used to construct the scenario. 


The scenario is modified from Raffetto’s SV04 base scenario to eliminate the red 
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infantry elements and include high value targets such as the Air Defense Artillery 
(ADA) elements, Surface-to-Air Missile (SAM) launchers, armor concentrations 
and Multiple Launched Rocket Systems (MLRS). To further evaluate the effects 
of cooperative sensing, communications models are incorporated into the 


network of UAVs, and various mixes of MAE and LAE UAVs are introduced. 


The model exploration uses a very efficient experimental design 
methodology to capture a large number of factors and their interactions which 
potentially affect the scenario outcomes. The controllable factors to be 
considered include UAV parameters (airspeed, reliability, detection/classification 
coverage and probability), network parameters (latency and degradation), ISR 
configurations (number of LAEs to one MAE), UAV behaviors (reactivity to other 
UAVs’ Situation Awareness, reactivity to spontaneous enemy contacts) and 
airspace deconfliction rules (minimum tactical separation in time and space). The 
uncontrollable factors to explore, which are often ignored in other studies, include 
the enemy Time Critical Target's parameters (concealment, detection range and 


vulnerability duration) that characterize the shoot-and-scoot behaviors. 


The experiment consists of running the agent-based models over a large 
combination of design points (i.e., settings for the controllable and uncontrollable 
factors) and collecting the measures of effectiveness. Regression and other 
statistical analysis techniques are then used to analyze the data and to provide 


insights to the following questions: 
¢ What are good cooperative sensing network configurations? 
¢ Are there diminishing returns with increasing UAVs allocated? 
« What are good UAV responses in a cooperative sensing environment? 


¢ What are the effects of degraded communications and increasing 
latency? 


« What are the effects of UAV failure in the presence of cooperative 


sensing? 


Furthermore, the analysis allows identification of main effects and 
interactions which may have significance impacts on ISR effectiveness for a 
cooperative sensing network, thus providing areas of focus for more detailed 


experiments and analysis. 


One should be aware that the employment of a disparate suite of sensors 
is an important consideration for cooperative sensing. However, it is not the focus 
of this thesis to model detailed sensor performances against a wide spectrum of 
terrains and targets. The sensors in the agent-based model, though they 
consider factors of detection degradation imposed by the different terrain 
features, are subjected to the same degradation on each terrain feature 
irrespective of the types of sensors and targets. The range of sensors explored is 
broadly classified into two main categories: the stand-off wide area sensors on 
board the MAE and the close-in small area sensors on board the LAE. 


F. AGENT-BASED SIMULATION 


Reductions in military operating budgets and improvements in computer 
technology have driven the increasing use of simulations throughout the military. 
Simulations are being employed in a wide variety of military applications 
including training, mission rehearsal, system analysis, system acquisition and 
tactical decision aiding. Furthermore, high complexity and nonlinearities in 
combat, where outcomes are highly correlated to battlefield conditions and 
events, make it too complicated and costly - if not impossible - to explore a full 
range of possibilities for any useful study in field experiments. In most of these 
cases, simulations are used for initial exploration to bound complex problems 
and tease out areas of emphasis relevant to the questions at hand that can then 
be expounded more meaningfully using methods such as field experiments and 
live tests and evaluations. The mission of the Defense Modeling and Simulation 
Office (DMSO) is to support the war-fighter by leading a defense-wide team in 
fostering interoperability, reuse, and affordability of Modeling and Simulation 


(M&S), as well as the responsive application of these tools to provide 
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revolutionary war-fighting capabilities and improve aspects of DoD operations. As 
a catalyst organization for the DoD, DMSO ensures that M&S_ technology 
development is consistent with other related initiatives by performing key 
corporate level functions necessary to encourage cooperation, synergism, and 
cost-effectiveness among the M&S activities of the DoD Components (Defense 
Modeling and Simulation Office website, 2005). 


The traditional (and widely used) modeling technique in the DoD uses 
differential equations to calculate casualties and changes in the frontlines. These 
equations are called Lanchester equations and were originally published in 1914. 
Many modern simulation models use variants of the Lanchester equations to 
predict the attrition rates of opposed forces massed in parallel strips across the 
battlefront. According to a RAND study, “these models were developed when 
computers had much more limited capabilities, making it necessary to reduce the 
number of simulation entities and to use aggregation techniques” (Gonzales, et 
al., 2001). The aggregated Lanchester equation-based models are beginning to 
fall out of favor in some DoD agencies for several reasons. Tighe cites a paper 
by Battilega and Grange that shows that the equations “do not accurately model 
many historical battles” (Tighe, 1999). This is surprising given that many of these 
battles follow the massive force-on-force attrition warfare formula for which the 
equations are designed. With modern warfare increasingly reliant on C4ISR 
systems, the old models break down even further. Gonzales points out that the 
legacy models cannot model individual C4ISR effects, and can only be 
represented by adjusting parameters. He continues by highlighting the fact that 
these models cannot take into account how information is used to support 
command decision-making processes. This severely limits their utility for 
assessing many Information Superiority concepts, such as force synchronization, 
that may be enhanced or enabled by advanced C4ISR systems (Gonzales, et al., 
2001). In other words, what good is C4lISR when your forces’ movements are 


scripted? 
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These problems have been addressed by a new generation of models that 
attempt to model complex adaptive systems. Forces are made up of individual 
“agents” that are programmed to follow a rough set of rules. “The individual 
agents are then responsible for making their own decisions as to how they 
should prosecute the battle” (Tighe, 1999). Hence they are adaptive. While the 
rules that govern an individual agent may be simple, a collection of agents 
interacting with one another (and the synthetic environment) will exhibit complex 
behaviors. These agent-based models, which are better suited for the analysis of 
modern concepts such as Network Enabled Warfare and Effects Based 
Operations, have begun to catch on in the DoD M&S community (Bjorkman and 
Sheldon, 2002; North, et al., 2003). 


G. MAP AWARE NON-UNIFORM AUTOMATA (MANA) 


The benefits of agent-based simulation models are greatly harnessed by 
the combat researchers and analysts in the Marine Corps Warfighting Laboratory 
(MCWL). The MCWL’s Project Albert is the research and development effort 
whose goal is to develop the process and capabilities of Data Farming. Data 
Farming is a method to address decision-maker's questions by applying high- 
performance computing to relatively simple models in order to examine and 
understand the landscape of potential simulated outcomes, enhance intuition, 
find surprises and outliers, and identify potential options. Data Farming is the 
method by which potentially millions of data points are explored and captured. 
This process is made possible, in part, by the exploitation of High Performance 
Computing assets and methods, and the project is fully supported by the Maui 
High Performance Computing Center (MHPCC). The Project Albert modeling 
approach is achieved through the development of a suite of models, sometimes 
called distillations, to drive home the point that these models are produced as an 
intentional complement to the very highly-detailed simulations being developed 
and used within the DoD, which by the very fact that they are so highly-detailed 
and encumbered, do not permit the examination of a very wide range of 
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possibilities and outcomes. By virtue of distillations being much easier to run and 
understand, they are proving to be effective tools that help capture and 
scientifically reproduce the ideas of subject matter experts, such as those 
thinking about tomorrow's concepts, doctrine, and requirements. This suite of 
entity-based models allow for rapid and highly tailorable changes in entity 
characteristics and behaviors, quite amenable to, and intentionally designed for 
rapid, repeatable concept exploration. Project Albert develops a suite of modeling 
platforms, rather than a single realization of a model. This has the added benefit 
of allowing the robustness of observations across modeling platforms to be 
examined. Also, each model has inherent strengths and unique capabilities with 
regard to each aspect of modeling how entities think, decide, shoot, move, and 
communicate (Marine Corps Warfighting Laboratory Project Albert website, 
2005). 


The MANA software, developed by the New Zealand Defense Technology 
Agency, is one of the distillation modeling platforms extensively used by the 
Project Albert Team to answer real world questions. The authors of the MANA 
User Manual (Galligan, et al., 2004) present a powerful illustration, “...the world 
is far more complicated than Newton’s equations (laws of motion)... Therefore, to 
rely on models built “on a bedrock of physics” is to deceive ourselves. It is a myth 
that a more detailed model is necessarily a better model, because it is impossible 
to capture accurately every aspect of nature”. They further their arguments by 
noting that the non-linear nature of equations describing many real world 
phenomena makes them extremely sensitive to initial conditions. This means that 
even infinitesimal errors in describing the real world initial conditions will cause 
the model to make predictions that are almost uncorrelated with actual events. 
The illustration provides a clear motivation for the design of MANA as what they 


refer as a scenario-exploring model. 


MANA is in a general class of models called agent-based models (ABMs), 
which have the characteristics of containing entities that are controlled by internal 
decision-making algorithms. Hence, an ABM combat model contains entities 
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representing military units that make their own decisions as they react to their 
surroundings, as opposed to the modeler explicitly determining their behaviors in 
advance. Models built in MANA, like those built using other Project Albert 
modeling platforms, are sometimes called agent-based distillations because their 
common intention is to model the essence of a problem rather than to describe 
every aspect of a military operation. MANA is designed to explore key concepts 
such as Situation Awareness, Communications, Terrain Features, Waypoints and 
Event-Driven Personality Changes. In MANA, agents can exhibit a surprisingly 
wide set of behaviors as long as careful configurations are given to sets of 
parameters that determine the agents’ propensities, move constraints, basic 
capabilities (such as sensors, weapons, interactions and movement speed) and 


movement characteristics (Galligan, et al., 2004). 


The model is designed to analyze the value of things such as situation 
awareness, command and control, and the informational edge that enhanced 
sensors provide. These features are limited in those models which purport to be 
detailed, highly physics based and rigorous where these aspects of combat can 
only be represented within the model in a completely arbitrary way by the 
modeler (Galligan, et al., 2004). These strengths of MANA as an agent-based 
model ties in closely to Gonzales’ (2001) presentation about the limitations of 
DoD legacy models as discussed in Section F. These are also the strengths that 
attracted the author to selecting MANA as a modeling tool for the analysis of 


cooperative sensing. 


H. THESIS ORGANIZATION 


Chapter Il begins by painting the modified Sea Viking scenario which 
includes the area of operation and the two sided force deployments and 
objectives. The chapter also describes the model development and in particular 
the modeling parameters and techniques used to achieve the desired agent 


behaviors. 
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Chapter Ill covers a detailed discussion of the controllable and 
uncontrollable factors of interest in the analysis and their operational significance 
and tradeoffs. It also provides a brief contrast of various possible designs of 
experiments and concludes that a Nearly Orthogonal Latin Hypercube design is 


the most efficient choice for running the experiments. 


In Chapter IV, the data analysis process is presented and followed by a 
meticulous exploration of the simulation results using regression and statistical 


analysis techniques. 


Chapter V concludes the thesis with a presentation of operational insights 
and recommendations from the results of the analysis. It also presents 
suggestions for follow-on work that the author considers worthwhile exploring. 
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ll. SCENARIO AND MODEL DEVELOPMENT 


A. THE SCENARIO 


The IPB scenario prior to the Marine’s amphibious assault depicted in 
Figure 2 provides the basis for the study of the effectiveness of a cooperative 
sensing network. The area of operation extends across the Los Angeles and San 
Diego city areas represented in the map as light orange regions. The Red target 
types included in this scenario represent two main arrays of targets characterized 
by their deployment profiles, vulnerability constraints and ease of detectability. 


Edwards 


jonio 





Figure 2. IPB Scenario for Marine Amphibious Assault [Best Viewed in Color] 


Area 5 at the upper right corner shows the Twenty-nine Palms 
encompassing the Red objective and their senior leadership, and is protected by 
an armored battalion. Area 4 in the central region defines the deployment ground 
of the multiple launched rocket systems and the anti-aircraft artillery elements for 
the defense of the Red objective. These targets are commonly deployed in 


isolation in a small fighting unit. They are usually well-concealed and have limited 
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windows of vulnerability, which makes detection and prosecution of these targets 
very challenging. The three armor battalions concentrated in Areas 1, 2 and 3 
move along the axis as shown in the map to stage a barrier patrol line against the 
coastal assault. The armor concentrations represent target arrays that are 
commonly deployed in formations and are “noisy” from the electronic and visual 
detection perspectives, hence making them easier targets for detection and 
prosecution. Area 6 represents the Blue MAE and LAEs supporting the IPB 
operation for the MEB. Each LAE has a pre-assigned area of responsibility to 
provide classification of enemy contacts, while the single MAE sweeps through 


the whole area of operation to provide wide area sensor coverage. 


B. MODEL DEVELOPMENT 
1. Environment 


The terrain shown in the Figure 3 is created by Raffetto (2004). The dark 
yellow in the terrain depiction represents city areas; the blue regions signify 
bodies of water; the brown, dark and light green regions identify the desert, 
dense and sparse forested areas respectively; the light yellow lines show the 
highways and major road networks in the cities. The terrain features are 
associated with parameters that affect the movement speed, engagement 
effectiveness, and concealment of the agents. These values provide multipliers 
to compute the overall effective speed, kill and detection outcomes of the agents 
at every time step. The blue aircraft icons depict the Blue MAE and LAEs; the red 
icons represent the Red armored vehicles and time critical targets; the pink icons 


identify the neutral elements. 
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Figure 3. MANA Model for IPB Scenario [Best Viewed in Color] 


The battle-space extends across 32.6 to 35.1 degrees in latitude and - 
118.8 to -115.6 degrees in longitude, which equates to 160 nm in width and 150 
nm in height. The maximum spatial resolution that can be represented in MANA 
is 1000 by 1000 cells, which computes to an average of approximately 287 by 
287 meters in both length and width per cell for this scenario. The model is set to 
elapse 10 real-time seconds for every time step taken for the simulation. Since 
the total amount of time for the IPB operations takes four hours, the simulation 
terminates after 1440 time steps have elapsed. The conversions between MANA 


units and real-world units are tabulated in a spreadsheet as illustrated in Figure 4 


to facilitate subsequent unit conversions in the development of the models. 
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BATTLEFIELD SETTINGS 
Real World 
Latitude Longitude 


Lower Left 32.6 -118.8]degree 1 degree latitude = 
Upper Right 3S.4 -115.6|degree 1 degree longitude = 


Width 159.46|nm 
Height 150.00jnm 
MANA Number of Cells 
Xx 1000 
¥ 1000 
Conversion Matrix 
each cell is how many units... 


nm miles feet km meters 
1 1.1508 6076.1155 1.8520 1852 











0.18 968.87 0.30 295.31 
0.17 911.42 0.28 277.80 
X/IY é 0.18 940.15 0.29 286.56 





each unit is how many cells... 
6.2713 5.4496 0.0010 3.3862 0.0034 
¥: 6.6667 5.7932 0.0011 3.5997 0.0036 
X/IY 6.4690 5.6214 0.0011 3.4930 0.0035 <-- Used for conversion between actual and MANA 





each time step is... 
secs mins hours 
10 0.1667 0.0028 


each time unit is how many steps... 
secs mins hours 
0.1 6 360 |<-- Used for conversion between actual and MANA 








# time steps in real time... 
hours time steps 
1440 


Figure 4. Unit Conversions Between MANA Model and Real-World Units 





2. Time Critical Target 


Time Critical Targets (TCTs) are targets requiring immediate response 
because they either pose (or will soon pose) a danger to friendly forces, or they 
are highly lucrative, fleeting targets of opportunity with an extremely limited time 
window of vulnerability, the attack of which is critical to ensure the successful 
execution of the subsequent battle operations. The Red MLRS and ADA 
elements in the model, totaled 30, are classified as time critical targets as they a 
limited windows of vulnerability characterized by their shoot-and-scoot tactics. 
These targets are generally deployed in terrains where concealment to escape 
enemy’s detection is almost perfect. The TCT comes out of its “hiding place” to a 
nearby operational spot only when it is ready to fire a salvo or there is a need to 
reload its ammunitions, and then goes right back into “hiding”. The TCT only 
subjects itself to detection and engagement vulnerabilities during this limited 
period of vulnerability which generally last no longer than 30 minutes, hence 


acquiring and prosecuting such targets is challenging. 
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The TCT in the model is developed by creating a cycle of waypoints and 
altering the behaviors of the agents in each state. The agent moves from the 
current waypoint to the next during the RunStart state and stays in this state for a 
defined vulnerable duration which will eventually be data farmed. Once it reaches 
the next waypoint and the vulnerability duration elapses, the agents change into 
the ReachFinalWaypoint state where the concealment rate (another parameter to 
be data farmed) is applied to these targets to emulate the “hide” behaviors. The 
TCT is assumed to have the capability to detect an approaching UAV with its 
counter-detection sensor and evade detection by the UAV by setting the 
concealment rate in the EnemyContact state. The TCT loops through the cycle 
until it is detected and classified by a Blue UAV as shown in the state transition 


diagram of Figure 5. 








Reach 
Final 
Waypoint 


Set concealment 
Set speed to zero 
(no movement) 


Reach waypoint 
and vulnerable 
duration elapsed 


Detect an 
approaching UAV 


Hiding duration 
elapsed 











Detect an 
approaching UAV 


Set concealment to 
zero 
Set speed to move Hiding duration 


to next waypoint elapsed Set concealment 


Set speed to zero 
(no movement) 


Figure 5. State Diagram of a Time Critical Target Agent 


3. Armor Battalion 


The model consists of four armor battalions; each is implemented as a 
separate squad and comprises 40 medium tanks. The three coastal battalions 
are scripted to follow pre-defined routes along the road axes and the last 
battalion is setup to patrol around the parameter of Twenty-nine Palms. The 


aS, 


agents are set up with propensities (or weights, as commonly known in many 
agent-based models) to move towards the next waypoint and to move on terrains 


that are easy-going, in this case, to follow the road networks whenever possible. 


4. MAE and LAE UAVs 


Due to the considerably limited area of operation, it is operationally 
infeasible to devote more than one MAE like the Predator to support the mission. 
However, an amphibious strike force commander may have up to five LAEs like 
the Pioneer at his disposal. While the MAE provides wide area surveillance and 
reconnaissance for the entire area of operation with poor classification capability, 
an LAE can provide effective close-in classification of the enemy contacts within 
its higher resolution sensor range. The models characterize these UAV 
parameters by varying the detection and classification ranges and the 
corresponding probabilities. 


The UAVs are given a “weapon” to shoot at the targets with range and 
probability equal to the range and probability for sensor classification, 
respectively. This means that a target is “killed” when being classified to prevent 
repeated classification of the same target. The “weapon” is loaded with a very 


large amount of “ammunition” to ensure the ammunition is not depleted. 


To support the analysis of the effects of LAE failure on ISR effectiveness 
of the cooperative sensing network, one UAV killer agent is created to “shoot” 
and disable a single LAE after a pre-determined time from the beginning of the 
scenario. The UAV killer is randomly positioned in the area of operation and 
remains inactive in the Default state until it is time for an LAE to fail. It then 
switches to another state where the “weapon” is enabled and “fires” a single shot 
with a probability of kill equal to 1.0 at the nearest LAE, hence disabling that LAE. 
The MAE in the model is assumed to be 100% reliable over the four-hour IPB 
operation, as MAEs are generally designed for longer endurance missions and 


with longer mean-time-to-failure parameters. 


20 


The movements and interactions of the UAV are characterized by a set of 
propensities in the model. Each UAV’s responsiveness to either an incoming 
contact from another UAV or a newly sighted enemy contact determines its 
tendency to be diverted from its original route. The propensity for each UAV to 
move towards the next waypoint is set to 50. By changing the propensities to 
move towards enemy in the Agent SA properties, and move towards unknown in 
the Inorganic SA, the reactivity of the UAV to such targeting information can be 
explored. Additionally, the airspace deconfliction parameter which determines the 
spatial separation of the UAVs in flight is varied by changing the propensities to 
move away from friends in the models. Further details on how these propensities 
are data farmed will be discussed in Chapter III. 


It should also be noted that the effectiveness of enemy contact detection 
and classification are very sensitive to the routes taken by the UAVs due to the 
relatively small UAV sensor coverage compared to the area of operation. If no 
UAV flies near a target, there will be neither a detection nor a classification. The 
routes for the LAEs are developed based on intuitive justifications after analyzing 
the terrain and target profiles. Refer to Appendix A for the design rationale and 
snapshots of the set of MAE and LAE routes. For the development of these 
routes, the author consulted with Captain Starr King, USN, NWDC-Sponsored 
Chair of Warfare Innovation and Chair of Applied Systems Analysis of the Naval 
Postgraduate School and Captain Kevin McMindes, a USMC pilot; both consider 
them reasonable and sufficiently detailed for the purpose of modeling and 
analysis to answer the questions of interest. However, one should not preclude 


other alternative routes based on various operational judgments and objectives. 


5. UAV Communications Network 


A communications network is set up in MANA to facilitate the sharing of 
information between the MAE and the LAEs, as illustrated in Figure 6. Two types 
of situational awareness maps are provided in MANA: a squad map which holds 


direct squad contact memory, and an inorganic map which stores contact 
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memories provided by other squads through communications links (Galligan, et 
al., 2004). The contacts discovered by the MAE are sent through the 
communications links and updated on the inorganic map of the LAE agents, and 
vice versa. The data links for both MAE and LAE are C-Band/UHF and are 
assumed to have communications ranges sufficient for coverage throughout the 
entire area of operation as long as there is Line Of Sight (LOS) between the 
UAVs. Although modern communications technologies offer superb link quality 
and availability from a technical view, communications degradation is, none the 
less, an important consideration in most military networks in which the systems 
are deployed due to the adverse operating environments and weather conditions. 
Hence, factors related to communications degradation are considered in the 
models by changing the values corresponding to message delay and link 


reliability over each communications link. 


MAE 





UAV te 
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Figure 6. Information Flow Between the MAE and the LAEs in the UAV Network for 
Shared Situation Awareness 


In actual operational environments, message latency represents the delay 
from the time a message is sent from the source UAV to the time it is received at 
the destination UAV, and varies depending on the distance between the two 
nodes and the current loading on the network. The link reliability in the model 
emulates the availability of the communications links for transmissions. There 


could be times where the LOS between two UAVs for transmission is not 
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available, in particular, when they are flying at low altitudes across rough terrains. 
Another major threat to link reliability could be the enemy’s act of electronic 


attack, which is a prevalent form of interdiction in modern warfare. 


C. MEASURES OF EFFECTIVENESS 


The DMSO defines a Measure of Effectiveness (MOE) as “a qualitative or 
quantitative measure of the performance of a model or simulation or a 
characteristic that indicates the degree to which it performs the task or meets an 
operational objective or requirement under specified conditions” (Defense 
Modeling and Simulation Office Online M&S Glossary, 2005). This study 
measures the expected proportion of Red TCT and armor entities classified at 
the end of the four-hour IPB to assess the effectiveness of employing 
cooperative sensing to detect each type of targets. The MOEs distinguish 
between different target types; this could potentially provide insights to the 
decision makers about appropriate ways to apply different UAV tactics and 
behaviors depending on the target characteristics and their deployment profiles 
in the presence of a cooperative sensing environment. For instance, when an 
LAE establishes a new armor contact, it may be inclined to follow the contact 
knowing that armors tend to operate in formation. This could result in a higher 
classification rate. In contrast, following a TCT contact may not result in a higher 


classification rate since TCTs tend to deploy in isolation. 
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lil. DESIGN OF EXPERIMENT 


A. EXPERIMENTAL FACTORS 
1. Controllable Factors — Blue ISR Parameters 


The number of factors affecting a cooperative sensing network’s 
effectiveness is large, and each factor may have a wide variation of potential 
levels (i.e., settings). The factors are often confounded and some even exhibit 
conflicting interactions among them. For instance, it may appear obvious that a 
faster UAV is desirable because it can cover a larger sweep area in a fixed 
duration of time. However, in most sensor systems involving human operators, a 
minimum integration time is needed for a target to “remain visible” within the 
sensor field of view to be considered a positive detection. As the UAV travels 
faster, the probability of detection and classification decreases since the time 
available for the sensor to look at the target reduces. From an endurance 
standpoint, a faster UAV is generally less fuel efficient, thus has less endurance 
than a slower UAV. Another case for conflicting interactions between factors 
includes the optimal use of MAE and LAE UAVs. The MAEs, in general, provide 
wider sensor coverage than the LAEs due to their higher operational altitudes. In 
reality, higher altitudes are associated with reduced signal strengths, reduced 
angular resolutions, and increased interferences; all of these are culprits for 
driving down the probabilities of detection and classification. Similarly, it may 
appear desirable to have a wider Field Of View (FOV) in compensation for lower 
altitude flights to have an equivalent sweep area. Again, one would quickly end 
up sliding on the tradeoff curve between a wide FOV with low resolution and 
narrow FOV with high resolution, and imagery resolutions have direct impact on 
the target classification abilities. These illustrations clearly present the case that 
many of these factors are closely interrelated; the analysis of their interactions is 


not trivial. 
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Factors are classified as controllable in the simulation experiments if they 
represent action options to decision makers for the real-world problems. For this 
reason, controllable factors are also commonly known as decision factors. |n 
concert with cooperative sensing, the decision factors that potentially contribute 
to the effectiveness of the sensing network in performing ISR-related missions, 


thus having direct relevance to gaining insights in our analysis, include: 
Factors related to sensing network configurations 
¢« Number of LAEs to one MAE 
Factors related to UAV parameters 
- Airspeed 
¢ Detection range 
* Classification range 
¢ Classification probability 
Factors related to network parameters 
¢ Link reliability 
*« Message latency 


Factors related to UAV routing behaviors 


* Reactivity to friendly UAV SA versus tendency to follow a planned 


route 


* Reactivity to a newly sighted enemy contact versus tendency to follow 
a planned route 


Factors related to airspace deconfliction 


« Minimum separation between UAVs 
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The UAVs selected for the study are not intended to be homogeneous but 
broadly classified into two categories, namely the MAE and LAE. The 
simultaneous investigation over the two categories of UAVs allows one to identify 
factors which are deemed important to the questions to be expounded, therefore 
provides further design considerations for architecting a cooperative sensing 
network for operational deployment. The factors include UAV airspeed, ranges 
for detection and classification, and detection and classification probabilities. To 
ensure the model is valid, the factors are varied with ballpark values based on 
current to near-future published technologies. These materials can be easily 
found from open-source literatures and databases such as The JANES Defense 
databases, The UAV Forum and Federation of American Scientist websites. The 
MAEs in the model have operational airspeed ranging from 60 to 100 knots and 
sensor detection range from 5,000 to 12,000 meters. The LAEs have operational 
airspeed ranging from 40 to 65 knots and sensor detection range from 2,000 to 
7,000 meters. In the model, the classification range is set no farther than the 


detection range by taking values from 0.6 to 1.0 of the detection range. 


It should be obvious that the ability of a UAV to detect and classify ground 
targets is dependent on its sensor coverage. The sensor coverage for a specified 
FOV increases with the UAV flight altitude as shown in Figure 7. The MAEs in 
the model have an operational altitudes ranging from 20,000 to 35,000 ft above 
ground level (AGL) while the LAEs from 5,000 to 20,000 ft AGL. The explored 
range of sensor FOV stretches from 30° to 60°. Figure 7 displays the resulting 
range of possible sensor coverage considered in the study. As an illustration, a 
tactical LAE flying at 15,000 ft AGL carrying an EO payload with a 60° FOV has 
circular sensor coverage with radius 2,640 meters (i.e., 8,660 ft). If, on detecting 
an unknown target, the same sensor decides to close in and lower its altitude to 
10,000 ft AGL and switch the EO camera to a 30° FOV in order to classify the 
target, it then has a sensor coverage radius of 817 meters. 
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UAV Sensor Coverage vs Altitude Chart 
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Figure 7. Graph of UAV Detection Range As a Function of Altitude and Field of 
View [Best Viewed in Color] 


A complex and detailed representation that mimics a real world sensor in 
the model is viewed unnecessary, in the author’s opinion, for this analysis. It is 
not the intent of the study to focus on highly-detailed and cumbersome physical- 
based modeling to give point estimate results. Instead, a simple model facilitates 
rapid execution to tap the power of agent-based distillations to provide insights 
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on the outcomes for varying model inputs. In this regard, cookie cutter models 
are assumed sufficient for detecting or classifying targets; targets falling within 
the sensor coverage have equal chance of being detected or classified. The LAE 
classification probability has values from 0.3 to 1.0. The MAE classification 
probability takes on values from 0.1 to 0.8 of the LAE classification probability to 


ensure that the latter is always lower than the former for every simulation run. 


One of the objectives as part of the study involves finding good sensing 
network configurations to support swift maneuver and precision engagement by 
the MEB. In the model, a single MAE having large sensor coverage for effective 
initial detection is supplemented by multiple LAEs to enable effective close-in 
target classification. The number of LAEs employed affects how much area can 
be covered and how long it takes to complete the search. An upper limit of five 
LAEs has been selected for the study. Although Raffetto suggests that it is 
unlikely that the Marine Corps will have the resources and personnel to employ 
more than three UAVs simultaneously as a current standard operating procedure 
for a MEB size area of operation (Raffetto, 2004), the addition of a fourth and fifth 
LAE is envisioned for future ISR capability extension due to constant reduction in 
the cost of acquiring and manning UAVs. The additional LAEs allow evaluation of 
overall ISR effectiveness as the number of LAEs increases in a cooperative 


sensing environment. 


The communications network forms the essential backbone for sharing 
detection information between the MAE and the LAEs. Degraded 
communications will inevitably shortchange the full potential one can harness 
from a cooperative sensing network to achieve high ISR capability. As discussed 
in the preceding chapter, electronic attack is becoming more prevalent in modern 
warfare. Coupled with the adverse weather and environmental conditions that 
these UAV networks are likely to operate in, the transmission reliability and 
accuracy of such information over the network must be considered. Link reliability 
and message delay are considered in the model to emulate such degradation in 
the communications network. While link reliability takes on values from 0% to 
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100%, the latency at which messages are transmitted from one UAV to another 
varies from 0 to 10 time steps in MANA, which corresponds to 0 to 100 seconds 


in real time. 


The interactions and behaviors amongst UAVs within the network are 
expected to have significant impacts on the effectiveness of a cooperative 
sensing network. The responsiveness of a UAV to other UAV SA determines how 
much the UAV’s next course of action is influenced by incoming targeting 
information from the network with respect to its original course of action. In 
MANA, this is achieved by changing the movement propensities of the UAV 
agents in response to an unknown contact in the inorganic SA. The ranges of 
weightings for these propensities vary from 0 to 100, and the responsiveness to 
the other UAV SA is computed as: 


(Move toward unknown — Follow waypoint) / (Follow waypoint) 


In the model, the follow waypoint weight is fixed at a value of 50; hence the 
responsiveness to other UAV SA ratio amounts to a range of -1.0 (completely 


unresponsive) to +1.0 (completely responsive). 


The other parameter that affects the behavior of a UAV is its reactivity to a 
newly sighted enemy contact. In reality, this defines the ISR tactics that permit 
dynamic redirection of UAVs from their intended routes toward newly detected 
hostile targets. In MANA, this is implemented by varying the movement 
propensity of the UAV agent towards unknown targets with respect to its 
propensity to follow waypoints. As in the previous metric, the response to a newly 
sighted target is computed as: 


(Move toward enemy — Follow waypoint) / (Follow waypoint) 


The weight for moving toward an enemy ranges from 0 to 100, giving the 
responsiveness to a newly sighted enemy contact ratio of -1.0 (completely 


unresponsive) to +1.0 (completely responsive). 
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The UAVs in the scenario are separated in time and space. This models 
the tactical airspace control rules that necessitate UAV controllers to plan for 
airspace deconfliction in any UAV operations. The analysis investigates the 
effects far and close airspace separations have on the effectiveness of 
cooperative sensing by modifying the UAV agent’s propensity to move toward or 
away from uninjured friends in MANA. A value of 0 corresponds to closer 
allowable airspace separation, while -100 leads to farther separation. These 
values represent the agents’ propensities to move toward or away from one 


another, and not the actual minimum separation distances between them. 


2. Uncontrollable Factors — Red Target Parameters 


Just as controllable factors are important in gaining insights about how 
they impact the ISR _ effectiveness of a cooperative sensing network, 
uncontrollable factors, which are often overlooked by operational analysts and 
decision makers, are of equal importance in this analysis, particularly since 
simulation is employed. In a mathematical modeling activity such as simulation 
one does, after all, get to control everything whether or not it is actually 
controllable in the real world. By exploring over ranges of settings for the 
uncontrollable factors, also known as noise factors, one could benefit by 
observing the influences these factors have on the system outcomes. These 
insights make the analyst more capable of deriving settings for the controllable 
factors that make the system more robust to a wider background variation, 
instead of one that optimizes decision factors after fixing each noise factor to a 
single value which is likely to deviate from reality. The robust design approach 
was first advocated by Taguchi for quality planning and engineering product 
design activities, and focuses on minimizing variation and sensitivity to noise 
(Taguchi and Wu, 1980). Sanchez (2000) suggests the following benefits of 
using a robust design approach for simulation: 


¢ Fewer surprises when decision is implemented 
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¢* Improved communication between analyst and “client” via expected 


loss 


* Ability to evaluate trade-offs between noise reduction costs and 


performance quality 


* Facilitates continuous improvement by seeking to reduce variability of 


response, not just achieving targeted value, and 


¢ Insights gained allow simultaneous improvement in performance and 


reduction in costs 


In the case of this analysis, valuable insights pertaining to the robustness 
of the cooperative sensing network can be achieved by varying those factors 
affecting the behaviors of the enemy’s forces. In particular, the TCT’s shoot-and- 
scoot tactics are perceived to have significant impact on the UAVs’ detection and 
Classification capability. The set of target parameters that contributes to its 
susceptibility to UAV detection include: 


* Concealment factor 
¢ Vulnerability time window 
¢ Sensor range for detecting nearby UAVs 


The target concealment factor has a range of 0% to 100%. The latter 
signifies that the target is completely invisible to passing UAVs; the former adds 
no additional reduction factor to the detection probability other than those already 
imposed by the terrain. The vulnerability time window describes the time duration 
for which a TCT exposes itself as it moves from its hiding position to its firing 
position or next deployment, thereby being vulnerable to UAV detections. The 
vulnerability duration varies from 10 to 30 real time minutes or 60 to 180 MANA 
time steps. The TCTs are assumed to have the capabilities to detect incoming 
UAVs and take positions for concealment thus reducing the signatures 
susceptible to detection. Whether the target can successfully take cover depends 
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largely on its sensor range for detecting nearby UAVs to provide sufficient 
response time for concealment. The TCT onboard sensor range varies from 0 to 


10 kilometers. 


One’s intuition suggests the reliability of UAVs may significantly impact the 
robustness of a cooperative sensing network. In addition to the base case, the 
overall classification effectiveness is explored for situations when a UAV fails or 
crashes, thus becoming unavailable in the IPB process. As the MAE in the 
sensing network is a critical node connecting and coordinating the ISR efforts 
amongst the LAEs, it is assumed in this study that the MAE has a long mean 
time to failure and the likelihood of its failure during the four hour IPB operation is 
negligible. Instead, the study expounds on the degree the ISR capability is 
degraded should a single LAE fail after some time into the operation, and 
evaluates how cooperative sensing could mitigate the “hole” in ISR plan. The 
model emulates a random failure to one of the LAEs at the zero, first, second, 
third and fourth hour. It should be noted that an LAE failure at the beginning of 
the IPB is nearly equivalent to the configuration of having one less LAE from the 
start, but will have a different and varied set of UAV routings since the failed LAE 
is chosen at random. Also, an LAE failing at the fourth hour has no effect on the 
classification outcome for the IPB since it fails right at the end of the scenario. 
This is a computationally efficient way to obtain some insights about the effects 
of LAE reliability without having to go into the details of modeling UAV 


breakdowns. 


The ranges of factors in both real world and MANA representations are 


depicted in Figure 8. 
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REAL WORLD & MANA METRICS 


Sensing Network Config 

A | Be 
UAV Parameters 
MAE Airspeed 
LAE Airspeed 
MAE detection range MAE Detect Range 
LAE detection range LAE Detect Range 
MAE classification range MAE Class Range 
LAE classification range LAE Class Range 
MAE classification probability |MAE Class Prob 
LAE classification probability }|LAE Class Prob 
Network Parameters 

Message latency Comms Latency second 0 

UAV Routing Behaviors 
Reactivity to friendly UAV SA 
versus tendency to follow a 
planned route React Inorg SA 
Reactivity to a newly sighted 
enemy contact versus 
tendency to follow a planned 
route React En Contact 

Airspace Deconfliction 
JUAV airspace separation _|Move Toward Friend | - | far near -100 

TCT Parameters 

Concealment factor TCT Conceal percent 
Vulnerability time window TCT Vul Duration minute 
Sensor range for detecting 
UAV TCT Detect Range km 





Figure 8. Experimental Factors and Levels in Both Real World Metrics and MANA 
Representations 


B. ORTHOGONAL LATIN HYPERCUBE DESIGN 


In simulation, experimental design provides a way of deciding before the 
runs are made which particular configurations to simulate so that the desired 
information can be obtained with the least amount of simulating. Carefully 
designed experiments are much more efficient than a “hit-or-miss” sequence of 
runs in which one simply tries a number of alternative configurations 
unsystematically to see what happens. Factorial and fractional factorial 
experimental designs are particularly useful in the early stages of 


experimentation, when one is pretty much in the dark about which factors 
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(parameters under investigation which are deemed to contribute to the effects of 
an experimental outcome) are important and how they might affect the 
responses. As the model’s behavior is gradually understood and a set of more 
specific goals are determined, a whole variety of specific experimental 
techniques can then be used to seek optimal combinations of factor levels that 
maximize understanding toward these goals (Law and Kelton, 2000). 


The full factorial experimental design quickly becomes unmanageable 
because the number of simulation runs needed escalates exponentially with 
increasing number of factors and factor levels. In this exploration of the use of 
multiple cooperative UAVs for ISR missions, the total number of factors in 
consideration is 19 with some factors having as many as 10 levels. It works out 
that 10'° (10 quintillion) design points are needed to study all possible factor-level 
combinations. If it is desirable to make 10 replications per design point, certainly 
a modest sample size from a statistical viewpoint, the total number of simulation 
runs would sum up to 107° (100 quintillion). If every replication takes a computing 
cluster one second to complete, the simulation would not finish even when the 


sun goes out and the earth dies; a problem one would not even think of solving. 


Fractional factorial designs provide a way to get good estimates of only 
the main effects and perhaps two-way interactions at a fraction of the 
computational effort required by a full factorial design. Basically, a 2* fractional 
factorial design is constructed by choosing 1/2° of all the possible design points 
and then running simulations for only these chosen points. Clearly, one would 
like p to be large from a computational efficiency viewpoint, but a larger p may 
also result in less information from the experiment, as one might suspect. 
Furthermore, the 2“ fractional factorial experimental design does not allow the 


investigation of non-linearity effects. 


A smarter design of experiment is desired. The Orthogonal Latin 
Hypercube (OLH) design was pursued for its efficiency, excellent space filling 
properties and design flexibility. Excellent space filling results in low correlation 


between input factors; design flexibility imposes few restrictions on the number of 
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factors and levels, and thus provides the ability to identify nonlinear relationships. 
The Nearly Orthogonal Latin Hypercube (NOLH) has nearly the same properties 
but a small amount of orthogonality is allowed in order to get better space-filling. 
A good space-filling design is one in which the design points are scattered 
throughout the experimental region with minimal unsampled regions; that is, the 
voided regions are relatively small. This means that the design points are not 
concentrated in clusters or solely at corner points of the region, as can happen 
with two-level factorial designs (Cioppa, 2002; Cioppa and Lucas, 2006). For a 
general discussion of designing simulation experiments, see Kleijnen et al. 
(2005). 


To construct the NOLH for this analysis, we used a Microsoft Excel 
spreadsheet created by Professor Susan Sanchez (Sanchez, 2005). The NOLH 
design considers 18 factors with 129 levels for each factor. Figure 9 is an extract 
of the NOLH design that depicts the first 50 of the 129 design points. 
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low level 0 0 0 0 0 0 0 108 72 17 7 
high level 10 100 100 100 100 100 100 180 117 42 24 
decimals 0 0 0 0 0 0 0 0 0 0 0 
1 [2] 13] (41 [5] [6] 7 [8] [9] [10] [11] [12] [13] [14] 
Comms Comms MAE LAE MAE Re- LAE Re-  Air-space MAE LAE MAE Detect LAE Detect MAE Class LAE Class LAE Class 
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tolnorgan to Inorgan to New to New Range Range 
factor name SA Sighting Sighting Ratio Ratio 
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Figure 9. Nearly Orthogonal Latin Hypercube of the Design of Experiment 


Figure 10 plots the design points derived from the 18 factor NOLH crossed 
with a single-factor, five-level design for the number of LAEs to each MAE to 
yield a total of 645 design points. An individual dot on each grid represents a 
design point with levels corresponding to the factors denoted on the row and the 
column. The pairwise scatter plot shows good space filling properties. The design 
points are highly orthogonal (all correlation values lower than 0.05), with the 
exception of the LAE classification and detection range pair, the MAE 
classification and detection range pair, as well as the LAE and MAE classification 
probability pair. These pairs of factors are deliberately constrained such that one 
factor is expressed as a random fraction of the other for every design point while 


creating the NOLH. For instance, the LAE classification range is restricted to a 
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varying fraction of the LAE detection range since it is illogical to have the former 
parameter larger than the latter for the same sensor system on an LAE. The 645 


design points are run for 50 replications each, which results in a total of 32,250 


MANA runs. 
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Figure 10. Pairwise Scatter Plot of Design Points Using a Nearly Orthogonal Latin 
Hypercube Crossed With a Single-Factor 
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The Tiller is a software tool in the Project Albert’s toolkit that allows the 
setup of XML study files comprising a large number of design points based on 
the experimental design. The study files provide the sequence of changing 
values for the associated MANA parameters, thus enabling batch processing 
without the need to load each simulation run manually. The study files, together 
with the base case simulation model file, are then submitted to the computing 
clusters in the MAUI High Performance Computing Center (MHPCC) hosted by 
the Air Force Research Laboratory Center, and managed by the University of 
Hawaii (Maui High Performance Computing Center, 2005). 
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IV. DATA ANALYSIS 


A. DATA PRE-PROCESSING 


The output data generated from the MHPCC is in either the Comma 
Separated Value (CSV) or the Microsoft Data Base (MCB) file format. The data 
file contains the excursion number for each design point and a random index 
representing the replication number in each design point, as well as the settings 
for factors that vary in the NOLH design. Each row in the data file includes the 
number of Blue and Red killed, the numbers of agents injured and killed per 
squad, the time steps for completing the simulation run, and a few other values. 
The only data of interest in the output files are the total number of Red killed and 
those relating to squads corresponding to the TCTs and armor battalions, and 
obviously, those parameter values that are varied in each simulation run. These 
data are then imported into the JMP IN Statistical Software (JMP Statistical 
Discovery Software website, 2005) on which data analysis is performed. The 
JMP Statistical Software provides many valuable analytical tools with a superior 
graphical user interface so there is a relatively short learning curve in using the 
basic features sufficient for most analytical use. The software is also selected for 
its powerful data pre-processing capability and its ability to handle extremely 
large data sets. 


The number of Red agents killed represents the number of Red targets 
that are classified by the Blue UAVs at the end of the simulation run. The 
expected proportion of targets classified for each target type is derived by 
summarizing the data over the replications for each design point and dividing the 
mean number of targets classified by their start-off total. These expected 
proportions of targets classified provide the MOEs for analyzing the effectiveness 
of the cooperative sensing network and the effects the factors in consideration 
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have on such sensing capability. The MANA parameters are also transformed 
back to their real-world representations before analyzing the data in order to 


ease interpretation of the results. 


B. ANALYSIS TOOLS AND METHODOLOGIES 


Several analytical tools and methodologies are applied to the model 
outputs. These sift out trends intrinsic within the data to help answer the 
questions of interest and provide useful insights and operational guidance 
relevant to the employment of cooperative sensing for military ISR purposes. The 
paragraphs below give brief descriptions of the main techniques used throughout 
the analysis. These techniques should by no means be used in exclusion of the 
others; using all in a complementary manner can provide better perspectives and 


operational insights. 


1. Data Partitioning 


Data partitioning is a form of exploratory modeling (sometimes known as 
data mining). It is a process of exploring a large amount of data, usually using an 
automated method, to find patterns and discoveries. Data partitioning is used to 
recursively partition a data set, automatically splitting the data at optimum points 
to maximize the difference in the values of the response variables between the 
branches of the split. The result is a decision tree that classifies each observation 
into a group (Sall et al., 2005). The technique is often used for exploring 
relationships without having a good prior model. It can handle large problems 


with relative ease and the results are very interpretable. 


2. Multiple Linear Regression 


Multiple Linear Regression is the technique of fitting or predicting a 
response variable from a linear combination of several other variables. The fitting 


principle is least squares, which finds a line through the data points that 
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minimizes the sum of squared distances to the line of fit (Sall et al., 2005). It is 
not the intend of this analysis to use the regression technique in the conventional 
way of deriving a mathematical model for predicting the proportion of Red targets 
classified given a set of cooperative sensing parameters; the agent-based 
distillation models are not set up for such details. However, the technique is used 
to provide valuable insights into the factors of significance, their interactions with 
one another, and the discovery of any non-linearities that might exist. Pareto 
plots are often used to visually plot bars emphasizing the order of importance of 
the factors from the most importance to the least. The prediction profiler in JMP 
IN presents a useful interface to show the predicted response for each 
combination of factor settings. It provides a handy way to look at the effect on the 
predicted response of changing one factor setting while holding the others 
constant, which can be useful for judging the contribution of each factor to the 


response. 


C. RESULTS OF ANALYSIS 
1. Sensing Network Configurations 


To investigate the impact the number of LAEs have on the proportion of 
targets classified, regression analysis is performed to obtain the best fit for the 
proportion of total Red targets, TCTs and armor targets classified against the 
number of LAEs respectively. Refer to the bivariate fits in Section 1 of Appendix 
B. The regression models show that the quadratic effects are significant for the 
proportions of total Red target and armor classifications (p-value < 0.001), but not 
for the proportion of TCT classification (p-value = 0.2870). 


For ease of illustration, the proportions of targets classified are grouped by 
the number of LAEs to obtain the average proportions of targets classified for 
each configuration with various numbers of LAEs. A quadratic curve is fitted for 
the proportion of total Red target classifications as shown in Figure 11; a line and 


a curve are fitted for the proportions of TCT and armor classifications 
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respectively, as depicted in Figure 12. An initial look at the results depicted by 
the curve in Figure 11 seems to suggest that one MAE and four LAEs is the most 
cost-effective cooperative sensing network configuration for the scenario in 
consideration. However, the detailed breakdown of the proportion of targets 
classified into the types of targets shown in Figure 12 reveals that while the 
classification rate for armor targets saturates when number of LAEs reaches four, 
the classification rate for TCT targets continues to increase with more than five 
LAEs. TCTs are less susceptible to UAV detections due to their higher 
concealment abilities. Armor battalions have strong detection signatures and are 
easily classified by relatively fewer LAEs, or even by the MAE itself which has a 
less superior classification capability. These imply that an investment of more 
sensing resources is required to detect TCTs. Another reason for the differences 
in the number of LAEs associated with diminishing returns could be due to the 
fleeing characteristics of TCTs and their ability to evade detections, thus a 
denser LAE concentration is required in the entire area of operation to defeat 
their counter-detection capability and limited windows of vulnerability. One would 
have completely missed these observations without looking at the detailed target 
classification proportions by types. This illustrates the importance of defining the 
correct set of MOEs. 


Bivariate Fit of Mean(Prop Red Class) By # LAE 
0.4 








o 
w 
a 


° 
w 


Mean(Prop Red Class) 
o 
i} ao 


oO 
= 
uo 





° 
ae 





# LAE 











—Polynoniial Fit Degree=2 





Figure 11. Quadratic Fit for the Average Proportion of Red Targets Classified Against 
the Number of LAEs 
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The sudden drop in the proportion of armor targets classified for five LAEs 
is found to be attributed to the distribution of routes. When the fifth LAE is 
introduced, it misses most of the armor elements in Area 1 as the convoy 
proceeds northward. This observation further demonstrates the sensitivity of the 
MOEs to variation in UAV routings, as anticipated during the model development 
phase and discussed in Chapter Il. 


Bivariate Fit of Mean(Prop TCT Class) By # LAE Bivariate Fit of Mean(Prop Armor Class) By # LAE 
0.125 0.3 











2 
1 

S 

Ny 

a 
L 


in(Prop Armor Class) 
So 
i 
I 


Mean(Prop TCT Class) 
Oo 
o 
S 
¢ 


° 
2 
a 

1 
Meai 
° 
a 

1 














0.025 : 1 1 1 0.1 1 
0 
































—— Linear Fit Polynomial Fit Degree=2 








Figure 12. Linear and Quadratic Fits for the Average Proportion of TCT and Armor 
Targets Classified Against the Number of LAEs 


The histogram plots depicted in Figure 13 allow further exploration of the 
proportions of TCT and armor targets classified for each MAE-LAE configuration. 
These reveal high frequencies at low classification rate for both types of targets, 
even though the means and the modes generally increase with the number of 
LAEs. Two categorical response variables are defined for the proportions of TCT 
and armor classification lower than 0.05 and 0.1 respectively. A logistic 
regression model is fitted for each of these response variables to investigate the 
rationales for the low classification rates. The details of the logistic regression 
models are shown in Appendix B. The analysis identifies LAE classification range 
and LAE responsiveness to MAE cueing as common factors of significance that 
contribute both to the low TCT and armor classification rates amongst few other 
factors. In the case of classifying TCTs, the TCT ability to conceal and its sensor 
range to evade UAV detection are important factors explaining the low 


classification rates. The partitioning analysis (refer to the partitioning tree in 
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Appendix B, Section 1) also suggests that LAE classification ranges less than 
3,200 and 2,600 meters can only classify an average of only 4% and 6% of the 


total TCT and armor targets, respectively. 
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Figure 13. Distributions of the Average Proportion of TCT (Top Row) and Armor 
(Bottom Row) Targets Classified for One LAE (Left) to Five LAEs (Right) 
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2. Factors of Significance 


The Multiple Linear Regression technique is used to determine factors that 
are significant to the proportion of targets classified during the four-hour IPB with 
the cooperative sensing network. Separate analysis are conducted for the TCT 
and armor targets, as distinctive differences exist between the two arrays of 
targets in terms of their target profiles and operational characteristics. 


Three stepwise linear regression models are fitted for the proportion of 
TCT classified with consideration of simply the main effects, the main and 
quadratic effects, and including the two-way interactions (Refer to Appendix B for 
the detailed models). The adjusted R-square values, instead of the standard R- 
square values, are considered when comparing the models to take into account 
the number of terms used to achieve the explanatory power of the models. The 
quadratic and interaction terms in the full model account for approximately 20% 
of explanatory power and are therefore included in the model for subsequent 
analysis. There are seven main effects, two quadratics and five interaction terms 
in the selected model as depicted in Figure 14. These fourteen terms explain 
73% of the total variability, based on the adjusted R-square value. 


Response Prop TCT Class 


Summary of Fit 


RSquare 0.731611 
RSquare Adj 0.725647 
Root Mean Square Error 0.030259 
Mean of Response 0.067764 
Observations (or Sum Wgts) 645 


Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 


Model 14 1.5723599 0.112311 122.6672 
Error 630 0.5768141 0.000916 Prob>F 
C. Total 644 2.1491740 <.0001 


Parameter Estimates 


Term Estimate Std Error tRatio Prob>|t| 
Intercept -0.062471 0.007717 -8.10 <.0001 
LAE React InOrg SA 0.0120452 0.002049 5.88 <.0001 
MAE Class Range 0.0000039 6.059e-7 6.50 <.0001 
LAE Class Range 0.0000212 9.586e-7 22.09 <.0001 
LAE Class Prob 0.0503922 0.005722 8.81 <.0001 
TCT Conceal -0.000549 0.000041 -13.40 <.0001 
TCT Detect Range -0.003615 0.000397 -9.11 <.0001 
#LAE 0.0198858 0.000842 23.60 <.0001 
(LAE React InOrg SA-0.00031)*(LAE Class Range-3569.77) 0.0000076 0.000002 4.38 <.0001 
(LAE Class Range-3569.77)"(# LAE-3) 0.000006 6.537e-7 9.19 <0001 
(LAE Class Prob-0.65039)*(# LAE-3) 0.0137356 0.004044 3.40 0.0007 
(TCT Conceal-50.0155)*(TCT Detect Range-5.08527) -0.000093 0.000016 -5.85 <.0001 
(TCT Conceal-50.0155)*(# LAE-3) -0.000156 0.000029 -5.39 <.0001 
(LAE Class Range-3569.77)*(LAE Class Range-3569.77) -7.013e-9 7e-10 -10.02 <.0001 
(TCT Conceal-50.0155)*(TCT Conceal-50.0155) -0.000008 0.000002 -5.05 <.0001 


Figure 14. Multiple Linear Regression Model for the Proportion of TCT Classification 


47 


The Pareto plot in Figure 15 ranks the factors in the full model in 


descending order of importance. The number of LAEs is listed as having the 


greatest impact on TCT classification rate. The prediction profiler in Figure 16 
highlights a substantial increase in the proportion of TCTs classified — from 0.047 
to 0.126 (approximately 2.5 times) — as the number of LAEs in the sensing 


network increases from one to five with all the other factors held constant. 
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Figure 15. Pareto Plot of Factors Significant to the Classification of TCTs 
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Figure 16. Prediction Profiler Showing Approximately 2.5 Times Improvement in TCT 
Classification By Increasing the Number of LAEs From One to Five 


It is also apparent that improving the LAE sensor qualities (from the 


aspects of classification 


range and classification probability) contributes 


considerably to the increase in the overall effectiveness of classifying the TCTs. 
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The MAE is generally inferior in its ability to classify targets due to the high flight 
altitude and wide sensor FOV in order to satisfy the wide area coverage 
requirement. In contrast, the LAEs have lower flight altitudes and more focused 
sensors that enable them to see targets with better resolutions, so though LAEs 
are myopic in their coverage, they have much better chances of classifying 
targets. The overall classification effectiveness of the cooperative sensing 
network increases when the LAEs are more responsive to the MAE cueing 
instead of following their pre-assigned routes. The interaction plot in Figure 17, 
however, shows an adverse effect when the LAEs have short classification 
ranges while being responsive to the MAE cueing. An LAE with small 
classification coverage, on being cued by the MAE, might take so long to fly 
close towards the TCT that it will miss its limited vulnerability window and lose 
the target completely. Even if the LAE arrives before the TCT again takes cover, 
a short classification range means the LAE has a limited number of classification 
opportunities, hence a low chance of successfully classifying the contact during 
its time on target. This wasteful trip can cost the LAE classification opportunities 
which would have arisen if it kept following its pre-assigned route. The results 
also show that extending the MAE classification range has a small but positive 
impact in the performance. This increases the number of classification 
opportunities during the MAE’s time on target and helps compensate for the poor 
Classification probability per attempt. Note that the benefits of having higher 
classification probabilities for LAEs are greater when there are more LAEs than 


when there are fewer. 


The negative coefficients associated with the TCT concealment and TCT 
detection range factors present the obvious case that TCTs which are well 
concealed with longer counter-detection sensor ranges are better able to evade 
incoming UAVs and avoid detection and classification. However, the interaction 
profiler (Figure 16) points out that the drop in the proportion of classification with 
increasing TCT concealment ability is more substantial for larger number of 
LAEs. On top of that, it is also observed that enhancing the two TCT’s 
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parameters simultaneously has a synergistic effect in reducing their susceptibility 
to UAV classification. This makes perfect sense, as it is not helpful for a TCT to 
be forewarned by its counter-detection sensor of an approaching UAV if it does 
not have an effective concealment capability. 
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Figure 17. Interaction Plot of Factors Significant to the Classification of TCTs 


Figure 18 shows the partitioning tree for the proportion of TCTs classified. 
The number of LAEs is the first break point for this model, which indicates the 
number of LAEs has a substantial impact on effectiveness; this is consistent with 
the results presented from the regression analysis. The tree displays a mean 
proportion of TCT classification of only 4.5% with fewer than four LAEs and 
10.2% with four or more LAEs (refer to Appendix B for the full partitioning tree). 
Moving down the right branch (i.e., using four or more LAEs), only 3.8% of the 
TCTs are classified if the LAE classification range is less than 2,866 meters. So it 
is futile to invest in more LAEs alone but it is important to invest in more LAEs 


that have sensors with longer classification ranges. Note, however, that despite 
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the investment one might devote into the cooperative sensing network, the 
increment in classification effectiveness against very well-concealed TCTs is 
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Figure 18. Right Branch of the Partitioning Tree for Proportion of TCT Classification 
With Four or More LAEs 


While partitioning trees are useful for exploring relationships without 
having a good prior model and inherently a powerful tool for capturing 
interactions, McMindes (2005) highlights a few drawbacks associated to such 
form of exploratory modeling. There is no sense of relative importance between 
the factors other than the hierarchical representation of the tree structure. For 
instance, Figure 18 shows the number of LAEs is more important than LAE 
Classification range, but how much more important? There is also no sense of 
indication of the sensitivity of a split. The LAE classification range is best split at 
2,866 meters, which is something absolutely not controllable in practice. So how 
much is lost if the splitting is shifted to the left or right by 50 meters? The break 
points also tend to be somewhat unstable, especially as the groups become 
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small. For these reasons, partitioning trees are used in conjunction with results 
from other techniques, such as Multiple Linear Regression, and to provide 
qualitative insights. 


A similar stepwise linear regression approach is again conducted for the 
proportion of armor targets classification. Three models were constructed using 
three sets of potential terms: the first model involved only the main effects, the 
second model involved both main and quadratic effects, and the third (full) model 
also included the two-way interactions. Here, the full model explains 80% of the 
total variability while the quadratic model already explains a notable 77%. The 
interaction terms account for less than 4% of the explanatory power in the full 
model, so the simpler model is chosen for subsequent analyses (refer to 
Appendix B for the detailed models). The seven main effects and two quadratic 
terms in the selected model are highlighted in Figure 19. These nine terms 
explain 77% of the total data variability. 


Response Prop Armor Class 


Summary of Fit 
RSquare 0.777112 
RSquare Adj 0.773953 
Root Mean Square Error 0.052919 
Mean of Response 0.192601 
Observations (or Sum Wgts) 645 


Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 


Model 9 6.1999396 0.688882 245.9965 
Error 635 1.7782373 0.002800 Prob>F 
C. Total 644 7.9781769 <.0001 


Parameter Estimates 


Term Estimate Std Error tRatio Prob>|t| 
Intercept -0.218018 0.019501 -11.18 <.0001 
LAE React InOrg SA -0.014634 0.003583 -4.08 <.0001 
LAE React En Contact 0.0154657 0.003583 4.32 <.0001 
LAE Airspeed 0.000569 0.000088 6.45 <.0001 
MAE Class Range 0.0000054 0.000001 5.14  <.0001 
LAE Class Range 0.0000567 0.000002 33.85 <.0001 
LAE Class Prob 0.0964538 0.010006 9.64 <.0001 
# LAE 0.0310841 0.001473 21.10 <.0001 
(LAE Class Range-3569.77)*(LAE Class Range-3569.77) -2.196e-8 1.187e-9 -18.49 <.0001 
(# LAE-3)*(# LAE-3) -0.022495 0.001245 -18.06 <.0001 


Figure 19. Multiple Linear Regression Model for the Proportion of Armor 
Classification 


The LAE classification range and the number of LAEs are ranked as the 


two most important factors that affect the proportion of armor classification, as 
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the Pareto plot in Figure 20 illustrates. The quadratics of the two main effects are 
next on the list, and indicate strong non-linear effects; the negative regression 
coefficients for the quadratic terms with positive corresponding main effect 
coefficients indicate that increasing either the LAE classification range or the 
number of LAEs will eventually lead to diminishing returns. This is illustrated 
clearly in the prediction profiler graphs in Figure 21. In fact, the proportion of 
armor classification shoots up from 0.053 to 0.311 (almost a_ fivefold 
improvement) as the LAE classification range extends from 1,433 to 4,900 
meters, and enlarging the LAE classification footprint beyond this upper limit 
reduces the overall classification effectiveness of the sensing network. Visually 
watching a few simulation runs suggests why this occurs. An LAE with long 
classification range is easily distracted by the surrounding neutral vehicles within 
the sensor footprint, and this renders the LAE ineffective in its mission to search 
out the Red targets. As discussed in the earlier section, the drop in the proportion 
of armor classification for five LAEs is attributed to the distribution of routes when 
the fifth LAE is introduced, since these routes miss most of the armor elements in 


Area 1. 


Pareto Plot of Transformed Estimates 
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Figure 20. Pareto Plot of Factors Significant to the Classification of Armor Targets 
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Figure 21. Prediction Profiler Showing Almost 5 Times Improvement in Armor 
Classification By Increasing LAEs Classification Range Less Than 3.5 Times 


As in the case of TCT classification, better quality LAE sensors (in terms 
of higher classification probabilities) enhance the classification effectiveness 
against the armor targets. Increasing the MAE classification range leads to a 
marginal improvement on the classification effectiveness as well, perhaps 
because it also increases the number of classification opportunities. In contrast, 
increasing the MAE classification probability was not shown to have an impact. 
The probability of an MAE successfully classifying an intended target in each 
opportunity is too low to make significant contribution to the overall effectiveness 
even at the highest range, and therefore not a significant factor. To find more of 
the highly mobile armors that are commonly deployed in a formation, a faster 
LAE does contribute to a higher classification success. This is not true for 
detecting the TCTs in this scenario, as they generally move over shorter 
distances. 


Unlike classifying TCTs, an LAE should lean towards being more 
responsive to follow a newly classified armor element since the discovery of one 
is likely to lead to the entire formation. Such LAE behaviors would not benefit 


TCT classification since they are mostly sparsely deployed in isolation. Being 
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responsive to the MAE cueing, in this case, may result in lower proportion of 
armor classification, but higher TCT discovery. 


The partitioning tree for the proportion of armor classified (Figure 22) 
shows LAE classification range to be the first break point for this model, which is 
consistent with the results presented from the regression analysis. The tree 
presents a mean proportion of armor classification of 23.2% when the LAE 
classification range is at least 2,579 meters and 5.8% below that. The next two 
levels of partitioning are based on the number of LAEs on both branches, thus 
indicating a non-linear relationship between the number of LAEs and the 
response variable with the best values around three to four LAEs. As discussed 
earlier, the introduction of the fifth LAE misses most of the armor elements in 
Area 1 due to its pre-assigned routing. A faster LAE could, however, mitigate this 


by reaching the area of responsibility earlier before the armor battalion moves 
away. 
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Figure 22. Partitioning Tree for Proportion of Armor Classification 


3. UAV Behaviors 


The contour plots in Figure 23 depict the contours for the proportions of 
TCT and armor classification with respect to the LAE responsiveness toward new 


enemy contacts they make (labeled “LAE React En Contact” on the Y-axis) and 
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MAE redirections (labeled “LAE React Inorg SA” on the X-axis). The dark blue 
patches represent higher proportions of classification in contrast to the light blue 
patches. The points on the plots indicate combinations of these two factors that 
appeared among the experiments. 


The plot shows a higher concentration of dark blue patches on the right 
half of the plot, thus suggesting better classification effectiveness against the 
TCTs when the LAEs have better responsiveness towards the MAE cueing. 
However, LAE response to a newly classified enemy contact does not seem to 
matter in achieving better overall classification performance. The result is intuitive 
since the TCTs are mostly sparsely distributed in this scenario. It is more likely 
that they are first detected by the MAE with the wide area sensor which then 
cues the LAEs to perform a close-in classification of these targets. Since the 
TCTs are sparsely located, having the LAEs follow newly discovered contacts is 
unlikely to lead to another TCT detection. 
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Figure 23. Contour Plots Showing Proportions of TCT (Left) and Armor (Right) 
Classification With Respect to LAE responsiveness Toward New Enemy 
Contacts and MAE Redirections [Best Viewed in Color] 


For classifying armor targets, the dark blue patches clustering around the 
top left of the plot suggest that having LAEs that are more responsive towards a 
newly classified enemy contact and less towards the MAE cueing significantly 
improves the overall classification effectiveness against armor targets. As armor 


elements tend to appear in formations, following a newly classified armor 
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element is likely to lead to the discovery of more armor elements in close 
neighborhood. However, once an LAE has found a cluster of targets, being too 
responsive to the MAE cueing might distract the LAE from making its kill and 


result in a lower overall proportion of armor classification. 


4. Impact of LAE Failure 


To examine how an LAE failure during the four-hour IPB operation might 
affect the overall classification performance — and how a cooperative sensing 
network could mitigate this performance degradation — the scenario with four 
LAEs is modified slightly so that one LAE is randomly selected to fail. Successive 
sets of experiments are conducted where this LAE fails immediately from the 
start of the scenario, and at the first, the second, and the third, and the last hour 
of the operation. When an LAE fails, the other UAVs continue on with the pre- 
assigned routes unless otherwise influenced by their responses to UAV 
redirections; there is no re-planning of the UAV routes. A few other factor ranges 
are modified based on the results described earlier in this chapter. First, only two 
discrete levels are used for link reliability: O represents no cooperation and 100 
represents full cooperation. The ranges of reactivity to other UAV SA for both the 
MAE and LAEs are also modified to take on values between 0 to +1. This 
represents that the UAVs are at least as reactive to redirection as to following 
their pre-assigned routes. The new NOLH is then crossed with the LAE lifetime 
(i.e., number of hours before a randomly selected LAE fails) to obtain the new 


experimental design matrix. 


The graphs in Figure 24 show the average proportions of TCT and armor 
classification when an LAE fails at various time instances with and without 
cooperative sensing capability. The graphs again clearly illustrate that 
cooperation does help in classifying TCTs but not armor targets. However, for 
both target types, having a failed LAE significantly degrades classification 
performance — as much as 21% in the four-hour operation, comparing an LAE 


failing at the start and the end of the operation. Although the improvement gained 
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in classification proportion for TCTs is constant regardless of the time of LAE 
failure, an improvement of 0.012 from a base of 0.065 to 0.087 is worth noting. 
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Figure 24. Graphs Depicting Average Proportions of TCT and Armor Classification 
Against LAE Lifetime With and Without Cooperative Sensing Capability 


In contrast, the gentler slope in the armor classification proportion when 
the LAE lifetime drops from four to zero hours shows that less degradation in 
classification performance occurs when the UAVs are part of a cooperative 
sensing network. This indicates that although cooperative sensing does not help 
in the overall armor classification in this particular scenario, it may provide a 


more robust solution in some scenarios. 
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In conjunction with the earlier observations, one should note that while it is 
undesirable for an LAE to be responsive to MAE redirection for a while once an 
armor entity is found, but instead follow the new contact which will likely lead to 
the discovery of more armor targets, at other times it may be worthwhile to react 
to an MAE redirection before the LAE has even made its initial classification. This 
suggests that appropriate switching between these behaviors during different 
phases of an LAE operation is necessary. In summary, being responsive to MAE 
cueing may not always give the best overall classification results; reacting to a 
cueing request may lead to the classification of other potential targets but risk the 
opportunity of a greater payoff by remaining in its path. Prior intelligence of the 
enemy’s courses of action and deployments is important for deciding appropriate 
redirections of LAEs even in the presence of cooperative sensing. Following a 
fixed set of standard operating tactics regardless of the targets of interest and 
enemy’s courses of action is not an effective way to deploy a cooperative 


sensing network. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. KEY OPERATIONAL INSIGHTS 


The overall effectiveness of the cooperative sensing network for 
classifying TCTs and armor targets with four LAEs stand at relatively low mean 
percentages of 9.8% and 27.2%, respectively. The low classification proportions 
are attributed to the limited time frame the UAVs have for the IPB operation and 
the modestly large number of neutral vehicles. Depending on the setup of the 
UAV behaviors and sensor parameters, the UAVs may be distracted by this 
clutter around interesting targets, thus unable to complete the routes they were 
initially designated. However, based on the authors past experiences with 
intelligence collection experiments, the results fall within the respectable range 
from an operational standpoint, given the short-range optical sensory assets 
searching in a considerably large area of operations. The relative classification 
performance of approximately one TCT to three armor targets makes reasonable 
sense as the armor targets have unlimited durations of vulnerability and the 
movement of an armor battalion generally emits strong signatures which leads to 
easy detection. They are also closely clustered which means the acquisition of 
one tank gives away the others in the formation. However, the TCTs are direct 
opposites in their operations; they are well concealed, operate individually and 
have limited periods of vulnerability, thus making them hard targets to catch. 


Two out of the seven main effects filtered out as factors of significance by 
the regression analysis describe Red TCT characteristics that are beyond the 
control of Blue forces. These two factors are the TCT concealment factor which 
dictates their ability to conceal themselves to avoid UAV detection and 
classification, and the counter-detection sensor range which determines their 
ability to detect approaching UAVs and hence evade UAV detection and 
classification. While TCT counter-detection sensor range does not show a steep 
impact on the UAV classification performance — and can be overcome with better 


LAE sensors with longer classification ranges and higher classification 
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probabilities — well-camouflaged TCTs reduce the overall UAV classification 
capability substantially. This suggests that mounting an EO sensor alone on the 
LAEs is not an effective solution to the problem of locating well-camouflaged 
TCTs. Perhaps a more viable option for classifying TCTs in densely forested 
areas is to equip the LAEs with IR sensors as well, to allow the targets to be 
tracked based on heat emissions instead of solely on optical detection. In this 
way, the classification opportunities are no longer limited to the windows of 


vulnerabilities when the TCT appears. 


The increase in the LAE responsiveness to an MAE redirection versus its 
tendency to follow its pre-assigned routes has a marginal positive effect on the 
proportion of TCT classification in the scenario. The MAE, which is designed to 
provide global initial detection, cues all the LAEs to perform a close-in 
classification when it finds suspicious contacts. Once the first LAE to arrive has 
successfully classified the contacts, the modeling approach used in this thesis 
makes certain that all other responding LAEs are turned back. However, such 
LAE behavior deters effective overall classification performance on armor 
targets. Instead, an LAE should adopt the tactics to follow a newly classified 
armor contact which would likely lead to the acquisition of more armor elements 
in the formation. An LAE that is responsive to the MAE cueing may benefit the 
first successful armor classification. However, if this LAE is frequently distracted 
by subsequent MAE redirections, and the associated traveling times are long 
because the cues are dispersed, then it may spend time traveling between 
unclassified targets that could otherwise be more effectively used to acquire 


more armor elements clustering around the first. 


Communication parameters such as link reliability and message latency 
are notably absent from the regression models that depict factors of significance 
in a cooperative sensing network deployment. The author opines the observation 
does correctly reflect the actual environment in that the range of message 
latency between 0 to 100 seconds is short compared to the LAE flight time to the 
suspicious contact in response to an MAE redirection, thus does not impact the 
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overall classification effectiveness of the sensing network. The relatively low 
tempo of MAE redirections during the short four-hour IPB operation might also 
give the MAE sufficient opportunities for retransmissions even in the presence of 
poor link availability. Note also that the airspace separation between the UAVs 
does not play a significant role in determining the overall classification 
proportions of the targets. The separations between the MAE and the LAEs in 
practice are not a concern since they tend to fly at different altitudes. Similarly, 
the separations between the LAEs are of little concern in this scenario since their 
flight plans have already ensured no two LAEs are close to each other at any 
point in time. On an MAE redirection, all other responding LAEs are turned back 
after the first LAE to arrive has successfully classified the contact; the likelinood 
of having two LAEs closing into the same target is small. It is obvious in 
retrospect that airspace separation has little impact, if not none at all, on this 


particular sensing network performance. 


While much is discussed about the behaviors of the UAVs in response to 
the presence of shared SA in a cooperative sensing environment in the earlier 
observations, many of the factors that contribute directly or indirectly to the extent 
of overall coverage of the sensing network appear in the analysis as significant. 
Factors such as MAE classification range, LAE classification range and number 
of LAEs show general trends of improving classification capabilities for both 
target types as their values increase. However, the increase in classification 
proportion for increasing LAE classification range tapers off, and even tips 
downwards around the 5,000 meters range. In this scenario, the surrounding 
neutral vehicles easily distract an LAE with a large classification footprint. This 
renders the LAE ineffective in its mission to search out the Red targets. In 
addition, the overall classification performance of the sensing network is very 
sensitive to the routings of the UAVs, which suggests that it is worthwhile to 
devote much time and effort into collecting information regarding the enemy’s 
target profiles, position and terrain prior to preparing the UAV search paths to 
provide the greatest ground coverage in areas of suspected operation. The 
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analysis reveals that while much is invested into netting UAVs together for 
interaction and cooperation, the traditional emphasis of investing in building 
better quality sensors that see further and clearer is as important. 


Cooperative sensing does provide a more robust solution in this scenario 
in terms of the classification performance should an LAE fail or be shot down 
during the four-hour operation. Without cooperation, an LAE failure will result in a 
“hole” in the ISR coverage of the entire area of operation unless a spare LAE is 
dispatched. The detrimental effect is mitigated to a certain extent with 
cooperative sensing in that another LAE may be redirected to patch this “hole.” 
However, an LAE being responsive to an MAE cueing may not always give the 
best overall classification results; reacting to a cueing request may lead to the 
classification of other potential targets but risk the opportunity of a greater payoff 
by remaining in its path. Prior intelligence of the enemy’s courses of action and 
deployments is important for deciding appropriate redirections of LAEs even in 
the presence of cooperative sensing. Following a fixed set of standard operating 
tactics regardless of the targets of interest and enemy’s courses of action is not 


the way forward in employing a cooperative sensing network. 


B. RECOMMENDATIONS FOR FUTURE WORK 


As with all exploratory investigations, the thesis has led to more questions 
and identified other relevant areas of interest for follow-on work. The ensuing 


paragraphs recommend some possible aspects that merit further research. 


The analysis highlights the sensitivity of the models and the MOEs to 
variations in the UAV routings. While the routings in this research are based on 
an intuitive justification which was deemed appropriate and sufficient for the 
study by subject matter experts, other routings would also be justifiable with 
differing operational considerations and other prior intelligence on the adversary 
courses of action, targets and terrains. Designing an experiment that explores 


various routing guidance and options, based on a broader number of scenarios 
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with possible enemy configurations (deployment plans and courses of action) to 
plan a robust set of routes, could yield a more accurate sense of the design of a 


cooperative sensing network and a robust solution. 


Based on the findings from the analysis, the responsiveness of an LAE to 
an MAE redirection should vary depending on the current status and recent types 
of targets classified. For instance, an LAE which has recently made an armor 
classification might not want to respond to an incoming MAE cueing, but follow 
the newly classified contact which is likely to lead to more armor target 
acquisitions. On the contrary, an idling LAE flying on its designated flight plan or 
one that has just made a TCT classification might get better payoff by being very 
responsive to an MAE redirection. This prompts another area that is worth further 
exploration: using an agent-based simulation model to derive more “intelligent” 
means of cooperation, taking into account the LAE status and the type of recent 
classification. 


Although MANA has been commonly used for evaluating ISR-related 
problems in a number of studies, such as those presented in the Background 
Work Section of Chapter |, it is seldom used for modeling cooperative sensing 
where information passing across a communications network could influence the 
behaviors of the agents. There is currently no accurate way to verify the models, 
except by visual inspection of a few instances of the model and by the 
justification of the set of average classification proportions of the targets falling 
within ballpark ranges. Porting the models over to another agent-based platform, 
like PYTHAGORAS (another agent-based distillation model in the Project Albert 
suite of models), would be beneficial for cross-validating the results of this 
analysis. This cross-validation is particularly useful to ensure that certain 
complex behaviors are not modeled incorrectly, or that insights obtained from the 
analysis are not due to artifacts of a specific modeling platform. However, one 
should be cautious to avoid focusing on specific numerical comparisons based 


on the model output data since different simulations yield different set of results. 
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Instead, a cross-validation process would reveal whether the two sets of model 


outputs yield qualitatively similar insights. 


Another advantage of porting the models over to PYTHAGORAS would be 
to leverage the power and versatility of its sensor setup. An agent in 
PYTHAGORAS may carry up to three sensors, and each sensor may have 
different detection parameters with different detectability on various terrain 
features (Bitinas, 2004). This flexible sensor setup enables the exploration of a 
cooperative sensing network configured for multiple sensors, such as an EO and 
a FOPEN, on a UAV. The effects of having a mix of sensor-bearing UAVs in the 
area of operation could also be explored. The inclusion of FOPEN or even the 
multi-modal SAR sensors could facilitate exploration of the improvement in the 
overall classification performance of the sensing network. However, the 
downside of using PYTHAGORAS is its simplicity in modeling communications 
with the rudimentary broadcast mechanism. The broadcast mechanism does not 
permit the considerations of message latency and link reliability as implemented 
in the MANA models, but simply exchanges SA information among the UAVs. By 
studying similar models in MANA and PYTHAGORAS, an analyst could attempt 
to exploit the strengths of each specific modeling platform to gain better 
understanding of how a broader variety of sensors, cooperative information 


sharing, and UAV tactics influence the mission effectiveness. 


To gain deeper insights on the effectiveness of a cooperative sensing 
network for ISR operations, scenarios where multiple local LAEs operate 
independently in the absence of a coordinating component like the MAE should 
be investigated. A greater challenge is modeling the effects of UAV swarming, 
where the deployment of a larger concentration of small UAVs with myopic 
sensors operating in a distributed yet cooperative manner could be explored. 
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C. SUMMARY 


The success of a battle is increasingly reliant on more accurate and timely 
collection of the battle-space intelligence. Unmanned systems, and certainly 
UAVs, are widely used in today’s military ISR operations and will be more 
prevalent in the next generation military as war-fighting concepts such as 
Network Centric Warfare and Distributed Operation emerge. Tighter integration 
and tactics development to enhance cooperation between these intelligence 
collection assets is one of the current focuses among the military intelligence 


community, and the motivation for this thesis. 


The use of agent-based simulation is definitely appropriate to seek 
insights relating to the tactics and behaviors in using a cooperative sensing 
network for military ISR. The models are easy to construct and consume 
relatively less computing power than most conventional simulation models. They 
can be executed for tens of thousands of replications to exhaust many possible 
representations of real-life situations. By applying suitable experimental designs, 
one can move from a realm of impossibility when the number of variations to be 
investigated is insurmountably large to a problem that can be realistically tackled 
within days or weeks. The simulation also provides ways to analyze a much 
wider variety of settings than time and cost would permit in a field experiment. 
The insights from agent-based simulation models may also provide guidance for 


setting up future higher-resolution simulation experiments or field tests. 


In this thesis, the author has built a number of agent-based models 
intended to capture the essential details of UAV performance for the purpose of 
answering the questions that might be of interest for system designers and 
operators to implement a cooperative sensing network for the military. However, 
with careful considerations, appropriate assumptions are also made to keep the 
model manageable within the identified scope. The data collected from the 
simulations are analyzed and investigated using a wide array of complementary 
analytical tools and techniques. The study has successfully led to several 


operational insights pertaining to the design and use of cooperative sensing for 
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ISR purposes, including the importance of having good UAV sensor capabilities 
and “intelligent” application of UAV cooperation tactics based on the interested 
target characteristics, to achieve high overall effectiveness. The study has also 


spawned further questions and identified relevant areas for follow-on work. 
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APPENDIX A. UAV ROUTINGS 


This appendix presents snapshots of the UAV routings for each MAE-LAE 
configuration as well as the rationales that led to these routings. During the 
development of these routes, the author consulted with Captain Starr King, USN, 
NWDC-Sponsored Chair of Warfare Innovation and Chair of Applied Systems 
Analysis of the Naval Postgraduate School and Captain Kevin McMindes, a 
USMC pilot. Both consider the routings reasonable and sufficiently detailed for 
the purpose of modeling and analysis to answer the questions of interest. 
However, one should not preclude other alternative routes based on various 


operational judgments and objectives. 
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1. MAE ROUTING 


The routing of the MAE is planned with the operational objective to provide 
a detection coverage that is as extensive as possible over the entire area of 
operation. The MAE performs the classic outward spiral search pattern by 
following the waypoints as depicted in Figure A-1. Note that the agents in MANA 
always start their movement from the waypoint with the largest index and 
proceed to the one with index zero in a reverse numerical order. In the models, 
the MAE is intended to loop around the waypoints such that it flies back to the 


starting waypoint once the last one is reached. 





Figure A-1_ Routing of the MAE 
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2. ONE-LAE ROUTING 


In the One-LAE configuration shown in Figure A-2, the one and only LAE 
is allocated to provide surveillance around the parameter of the Red objective, 
Twenty-nine Palms, where critical Red activities are deemed to be of most 
interest to the MEB commander. 





Figure A-2. Routing of One LAE 


ra 


3. TWO-LAE ROUTINGS 


On availability of a second LAE, it is assigned to provide coverage over 
the larger of the two cities, Los Angeles and its suburbs, while leaving the first 
LAE to continue its surveillance of Twenty-nine Palms. The only intuitive 
justification for giving priority to the cities over the forested and mountainous 
areas when allocating areas of responsibilities to the LAE assets is based on the 
assumption that the developed areas are considered to be more accessible by 
well-developed road networks and infrastructures that facilitate deployment of 
large Red forces. The routes of the two LAEs are shown in Figure A-3. 





Figure A-3 Routing of Two LAEs 
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4. THREE-LAE ROUTINGS 


With the third LAE, it is desired to extend the ISR coverage to the next 
larger city, San Diego, while focusing LAE assets on the regions that are in a 
direct path from the amphibious landing to the Red objective. The routings are 
shown in Figure A-4. The area of coverage may appear ambitious for a local 
sensor capability as an LAE, but it is considered important to establish situational 
awareness in these regions where many Red air defense elements and time 
critical targets are expected to operate to defend Twenty-nine Palms against the 


Blue attack. 





Figure A-4 Routing of Three LAEs 
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5. FOUR-LAE ROUTINGS 


The addition of the fourth LAE provides more effective ISR coverage of 
the Los Angeles loop whose area of responsibility is initially very large for a 
single LAE. The original area of responsibility is now broken into two halves: the 
western and eastern loop, each allocated to an LAE as illustrated in Figure A-5. 





Figure A-5 ~=Routing of Four LAEs 
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6. FIVE-LAE ROUTINGS 


With five LAEs, the large San Diego loop is now divided into a northern 
and southern loop as depicted in Figure A-6. A single LAE is responsible for the 


ISR coverage of each loop. 





Figure A-6 Routing of Five LAEs 
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APPENDIX B. LINEAR REGRESSIONS, PARTITIONING TREES, 


GRAPHS AND PLOTS FOR ANALYSIS 


This appendix contains the linear regression outputs, partitioning trees, 
prediction profiler graphs and other plots used in the complete analytical effort. 
The materials are arranged according to their relevance to the analytical focus 


provided by the initial questions of interest. 


1. SENSING NETWORK CONFIGURATIONS 


Distributions of the proportion of total red targets classified for one 
(leftmost) to five (rightmost) LAEs show a general trend of better average 
classification results as the number of LAEs increases. Note that poor results can 


occur for all situations. 
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Distributions of the proportion of TCT classified for one (leftmost) to five 


(rightmost) LAEs show a general trend of better classification results as the 


number of LAEs increases. Once again, poor outcomes can occur even when the 


number of LAEs is large. 


Prop TCT Class 


Prop TCT Class 


Prop TCT Class 


Prop TCT Class 


Prop TCT Class 
































































































































0.3 0.3 0.3 0.3 0.3. 
0.25 0.25: 0.25: 0.25 0.25 
0.2 0.2 0.2 0.2 0.2 
0.15 0.15, 0.15: 0.15, 0.15 
0.1 0.1 0.1 0.1 iI 0.1 oO 
0.05 = 0.05 [ S 0.05 S 0.05 0.05 
0 0 0 [ 0 0 
Quantiles Quantiles Quantiles Quantiles Quantiles 
100.0% maximum 0.10467 100.0% maximum 0.16067 100.0% maximum 0.15800 100.0% maximum 0.26600 100.0% maximum 0.31733 
99.5% 0.10467 99.5% 0.16067 99.5% 0.15800 99.5% 0.26600 99.5% 0.31733 
97.5% 0.08433 97.5% 0.14767 97.5% 0.15117 97.5% 0.23400 97.5% 0.27850 
90.0% 0.05667 90.0% 0.11333 90.0% 0.10200 90.0% 0.18733 90.0% 0.18533 
75.0% quartile 0.03867 75.0% quartile 0.08533 75.0% quartile 0.06600 75.0% quartile 0.13933 75.0% quartile 0.15833 
50.0% median 0.02533 50.0% median 0.05000 50.0% median 0.04200 50.0% median 0.09733 50.0% median 0.11200 
25.0% quartile 0.01200 25.0% quartile 0.02800 25.0% quartile 0.02100 25.0% quartile 0.04100 25.0% quartile 0.04700 
10.0% 0.00467 10.0% 0.01133 10.0% 0.00933 10.0% 0.01467 10.0% 0.01333 
2.5% 0.00133 2.5% 0.00467 2.5% 0.00333 2.5% 0.00500 2.5% 0.00417 
0.5% 0.00000 0.5% 0.00333 0.5% 0.00200 0.5% 0.00133 0.5% 0.00067 
0.0% minimum 0.00000 0.0% minimum 0.00333 0.0% minimum 0.00200 0.0% minimum 0.00133 0.0% minimum 0.00067 
Moments Moments Moments } Moments Moments } 
Mean 0.0280413 Mean 0.0570749 Mean 0.0487442 Mean 0.097907 Mean 0.1070543 
Std Dev 0.0211298 Std Dev 0.0383017 Std Dev 0.0360301 Std Dev 0.062705 Std Dev 0.0714522 
Std Err Mean 0.0018604 ‘Std Err Mean 0.0033723 ‘Std Err Mean 0.0031723 ‘Std Err Mean 0.0055209 Std Err Mean 0.006291 
upper 95% Mean 0.0317224 upper 95% Mean 0.0637476 upper 95% Mean 0.0550211 upper 95% Mean 0.108831 upper 95% Mean 0.1195021 
lower 95% Mean 0.0243603 lower 95% Mean 0.0504023 lower 95% Mean 0.0424673 lower 95% Mean 0.086983 lower 95% Mean 0.0946064 
N 129 N 129 N 129 N 129 N 129 


Distributions of the proportion of armor classified for one (leftmost) to five 


(rightmost) LAEs show a general trend of better classification results as the 


number of LAEs increases. The exception observed for the scenario with five 
LAEs is attributed to the distribution of routes when the fifth LAE is introduced 


which misses most of the armor elements in Area 1 as the convoy moves 


northwards. 
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| Prop Armor Class J | Prop Armor Class { Prop Armor Class ] | Prop Armor Class Prop Armor Class | 
0.5 0.5 0.5 0.5 05: 
0.4 0.4 
0.3 0.3 
0.2 : [ - 0.2 S| 
H ce 
0.4 [ S| 0.4 
0 0 0 

Quantiles Quantiles | { Quantiles ( Quantiles | Quantiles | 
100.0% maximum 0.20400 100.0% maximum 0.31475 100.0% maximum 0.42825 100.0% maximum 0.47100 100.0% maximum 0.38138 
99.5% 0.20400 99.5% 0.31475 99.5% 0.42825 99.5% 0.47100 99.5% 0.38138 
97.5% 0.17284 97.5% 0.25678 97.5% 0.41066 97.5% 0.45581 97.5% 0.35828 
90.0% 0.15375 90.0% 0.23150 90.0% 0.36075 90.0% 0.40950 90.0% 0.29938 
75.0% quartile 0.12600 75.0% quartile 0.20531 75.0% quartile 0.31406 75.0% quartile 0.36756 75.0% quartile 0.26363 
50.0% median 0.10675 50.0% median 0.16350 50.0% median 0.27337 50.0% median 0.30650 50.0% median 0.22075 
25.0% quartile 0.07681 25.0% quartile 0.10681 25.0% quartile 0.18494 25.0% quartile 0.21044 25.0% quartile 0.12350 
10.0% 0.03625 10.0% 0.04350 10.0% 0.04400 10.0% 0.05050 10.0% 0.04825 
2.5% 0.01891 25% 0.02234 2.5% 0.03206 2.5% 0.02991 2.5% 0.02119 
0.5% 0.00762 0.5% 0.01087 0.5% 0.01050 0.5% 0.01038 0.5% 0.00775 
0.0% minimum 0.00762 0.0% — minimum 0.01087 0.0% minimum 0.01050 0.0% minimum 0.01038 0.0% — minimum 0.00775 

| Moments J | Moments J | Moments } Moments Moments 

Mean 0.1003576 Mean 0.1527238 Mean 0.24197 Mean 0.2716308 Mean 0.1963246 
Std Dev 0.0408488 Std Dev 0.067695 Std Dev 0.1110752 Std Dev 0.1271268 Std Dev 0.0929332 
Std Err Mean 0.0035965 Std Err Mean 0.0059602 Std Err Mean 0.0097796 Std Err Mean 0.0111929 Std Err Mean 0.0081823 
upper 95% Mean 0.1074739 upper 95% Mean 0.1645171 upper 95% Mean 0.2613206 upper 95% Mean 0.2937779 upper 95% Mean 0.2125147 
lower 95% Mean 0.0932412 lower 95% Mean 0.1409305 lower 95% Mean 0.2226193 lower 95% Mean 0.2494838 lower 95% Mean 0.1801345 
N 129 N 129 N 129 N 129 N 129 


The bivariate fits for the proportion of target classification by the number of 
LAEs show that the quadratic terms are statistically significant in the models for 
the total red and armor targets but not in the model for the TCT targets. 


(Bivariate Fit of Prop Red Class By # LAE } 
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0.7 ; [ Polynomial Fit Degree=2 } 
0.644 i : Prop Red Class = 0.1499145 + 0.0509699 # LAE - 0.0212292 (# LAE-3)'2 
! A . 
! ! | Summary of Fit 
054 : l } 
8 | RSquare 0.275092 
904- RSquare Adj 0.272834 
id 03 Root Mean Square Error 0.130754 
a : | Mean of Response 0.260366 
2 924 ' ' | Observations (or Sum Wgts) 645 
: 
o14 ' H : { Analysis of Variance } 
| | H Source DF Sumof Squares Mean Square F Ratio 
0 1 oe 7 = 7 rt af haa 5 Model 2 4.165259 2.08263 121.8147 
#LAE Error 642 10.976078 0.01710 Prob>F 
C. Total 644, 15.141337 <.0001 
—Polynomial Fit Degree=2 | Parameter Estimates | 
Term Estimate Std Error tRatio Prob>|t| 
Intercept 0.1499145 0.013552 11.06 <.0001 
#LAE 0.0509699 0.003641 14.00 <.0001 
(# LAE:3)42 -0.021229 0.003077 -6.90 <.0001 


Bivariate Fit of Prop TCT Class By # LAE 








T Polynomial Fit Degree=2 
Prop TCT Class = 0.0055754 + 0.0198858 # LAE + 0.0012658 (# LAE-3)'2 


o 
w 
1 


0.255 : Summary of Fit 
” : 
8 024 : : RSquare 0.238704 
2) ' ! RSquare Adj 0.236333 
6 0.154 : : i H 
pe” ! | Root Mean Square Error 0.050483 
S$ 014 Mean of Response 0.067764 
a Observations (or Sum Wgts) 645 
0.054 


Analysis of Variance 





























05 Source DF Sumof Squares Mean Square F Ratio 
i ee 2 a 3 em 4 NoGet 7 Model 2 0.5130171 0.256509 100.6496 
#iAE Error 642 1.6361569 0.002549 Prob>F 
C. Total 644 2.1491740 <.0001 
—Polynomial Fit Degree=2 Parameter Estimates 
Term Estimate Std Error tRatio Prob>|t| 
Intercept 0.0055754 0.005232 1.07 0.2870 
#LAE 0.0198858 0.001406 14.15 <.0001 


(# LAE-3)42 0.0012658 0.001188 1.07 0.2870 


Bivariate Fit of Prop Armor Class By # LAE 



































J ; Polynomial Fit Degree=2 
044 i i Prop Armor Class = 0.1443391 + 0.0310841 # LAE - 0.022495 (# LAE-3)42 
wa | ! : Summary of Fit 
oy ‘ 
Bos ! ! RSquare 0.270777 
5 H | RSquare Adj 0.268506 
5 va i | Root Mean Square Error 0.095195 
2 al : . Mean of Response 0.192601 
a 1 ‘ i Observations (or Sum Wots) 645 
O17 | | H 1 i Analysis of Variance 
5 1 
1 | t ! | ! Source DF Sumof Squares Mean Square F Ratio 
6 i rere 7 ae 3 ard, 7 os 7 Model 2 2.1603087 1.08015 119.1947 
#LAE Error 642 5.8178682 0.00906 Prob>F 
C. Total 644 7.9781769 <.0001 
—Polynonial Fit Degree=2 Parameter Estimates 
Term Estimate Std Error tRatio Prob>|t| 
Intercept 0.1443391 0.009866 14.63 <.0001 
#LAE 0.0310841 0.00265 11.73 <.0001 


(# LAE-3)42 -0.022495 0.00224 -10.04 <.0001 


A stepwise logistic regression model is fitted for the categorical response 
variable lowTCTclass to investigate the rationale for high frequencies clustering 


around the low value as depicted in the distribution graphs. 


1 if proportion of TCT classified < 0.05 


otherwise 


lowTCTclass = 


80 


Nominal Logistic Fit for Low TCT Class 
Whole Model Test 


Model -LogLikelihood DF ChiSquare Prob>ChiSq 
Difference 109.95633 6 219.9127 <.0001 

Full 337.11662 

Reduced 447.07295 

RSquare (U) 0.2459 

Observations (or Sum Wgts) 645 

Converged by Gradient 

Lack Of Fit 

Source DF -LogLikelinood ChiSquare 
Lack Of Fit 122 104.84869 209.6974 
Saturated 128 232.26793 Prob>ChiSq 
Fitted 6 337.11662 <.0001 

Parameter Estimates 
Term Estimate Std Error ChiSquare Prob>ChiSq 
Intercept -4.0189294 0.6504023 38.18 <.0001 
LAE React InOrg SA 0.48629692 0.1619628 9.02 0.0027 
MAE Detect Range 0.00019502 0.0000467 17.44 <.0001 
LAE Class Range 0.00087507 0.00009 94.48 <.0001 
LAE Class Prob 1.79752337 0.4586052 15.36 <.0001 
TCT Conceal -0.0241628 0.0033894 50.82 <.0001 
TCT Detect Range -0.138945 0.0320138 18.84 <.0001 


For log odds of 0/1 


The model reveals that the factors of significance for the low classification 
rates are LAE classification range and _ classification probability, LAE 
responsiveness to other UAV SA, MAE detection range, TCT concealment and 
TCT evasion detection range. 

The prediction profiler graph for the logistic regression model suggests 
that the LAE classification range is the most influential factor given that all others 


are kept constant at their nominal values. 


Prediction Profiler 





kK 1 
ot 
Bb 
zB 0.491424 
Sa 
xr? 
0 
- 1 aad N oO oO Tt oO = oO =] oO fo} 
0.00031 2 8454.64 3 ¥ 3600 8 S 0.65039 50.0155 = 5.08527 = 
LAE React MAE Detect LAE Class LAE Class TCT Conceal TCT Detect 
InOrg SA Range Range Prob Range 
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The partitioning tree for the proportion of TCT targets further suggests that 
an LAE with classification range less than approximately 3,200 meters has a 
mean classification proportion of only 0.04. Even if the LAE has better 


Classification capabilities, the TCT classification rate is even worse if the TCT 
concealment is above 85%. 


| 
All Rows 
R-square = 0.341 


Count 





645 
Mean 0.0677643 























Std Dev 0.0577687 
LAE Class Range<3152 LAE Class Range>=3152 
Count 245 Count 400 
Mean 0.0360762 
Std Dev 0.0297034 














Mean 0.0871733 
Std Dev 0.0620695 




















TCT Conceal>=86 
Count 





TCT Conceal<86 
60 Count 
Mean —0.0295444 
Std Dev 0.0264837 








340 
Mean 0.0973431 
Std Dev 0.0609992 


























LAE Class Range<4585 











LAE Class Range>=4585 
Count 200 Count 140 
Mean 0.08711 Mean 0.111969 
Std Dev 0.0529167 











Std Dev 0.0685559 























LAE React InOrg SA<-0.22 


LAE React InOrg SA>=-0.22 
Count 


40 














Count 100 
Mean 0.0817333 Mean 0.1240533 
Std Dev 0.0506466 StdDev 0.0712144 





A similar analytical approach is also conducted to investigate the rationale 


for the low proportion of armor classified. The categorical response variable 
lowARMORclass is defined as follows. 


ap APNIO Resse = ( if proportion of armor classified < 0.1 


otherwise 
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Nominal Logistic Fit for Low Armor Class 


Whole Model Test 


Model -LogLikelihood DF ChiSquare Prob>ChiSq 
Difference 197.73733 4 395.4747 <.0001 
Full 166.87878 
Reduced 364.61611 
RSquare (U) 0.5423 
Observations (or Sum Wgts) 645 
Converged by Gradient 
Lack Of Fit 
Source DF -LogLikelihood ChiSquare 
Lack Of Fit 124 73.35416 146.7083 
Saturated 128 93.52462 Prob>ChiSq 
Fitted 4 166.87878 0.0802 
Parameter Estimates 
Term Estimate Std Error ChiSquare Prob>ChiSq 
Intercept -10.494934 1.089704 92.76 <.0001 
LAE React InOrg SA -0.5623604 0.2333648 5.81 0.0160 
MAE Class Range 0.00027273 0.0000733 13.86 0.0002 
LAE Class Range 0.00267633 0.0002349 129.79 <.0001 
MAE Class Prob 5.50551346 1.0309848 28.52 <.0001 


For log odds of 0/1 


The model suggests that LAE classification range, LAE responsiveness to 
other UAV SA, MAE classification range and classification probability are 


significant determinants of the low classification rates for armor targets. 


The LAE classification range is again identified by the prediction profiler as 


the most significant factor given that all others are kept at their nominal levels. 


Prediction Profiler 


1 


1) 


0.053223 





P(Low Armor 
Class 


0 





0.00031 2 678413N Y 3569778 J 0.357759 
oO = = oO 
LAE React MAE Class LAE Class MAE Class 
InOrg SA Range Range Prob 


The partitioning tree shows that an LAE with classification range of less 


than 2,600 meters has a mean proportion of armor targets classified of about 


0.06. 
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All Rows 


R-square = 0.494 Count 645 


Mean 0.1926014 
Std Dev 0.1113035 














































































































LAE Class Range<2579 LAE Class Range>=2579 
Count 145 Count 500 
Mean 0.0576078 Mean 0.2317495 
Std Dev 0.0325195 StdDev 0.094086 
MAE Class Range<6304 MAE Class Range>=6304 LAE Class Range<2866 LAE Class Range>=2866 
Count 65 Count 80 Count 45 Count 455 
Mean 0.0348596 Mean 0.0760906 Mean 0.1548417 Mean 0.2393558 
Std Dev 0.0140108 Std Dev 0.0315735 StdDev 0.063788 Std Dev 0.0932255 
MAE Class Prob<0.32 MAE Class Prob>=0.32 
Count 205 Count 250 
Mean 0.2175262 Mean 0.257256 
Std Dev 0.0879724 StdDev 0.093764 

















2. FACTORS OF SIGNIFICANCE 


Next, Multiple Linear Regression models are provided for the proportion of 
TCT classified with consideration of only the main effects (top left), the main and 
quadratic effects (top right) and including the two-way interactions (bottom). 


Response Prop TCT Class 


Summary of Fit 
Response Prop TCT Class 


RSquare 0850438, 
Summary of Fit RSquare Adj 0.645179 *s 
RSquare S8TT2h., Root Mean Square Error **-Q.034g44°" 
RSquare Adj © 0.576524 a Mean of Response 0.067764 
Root Mean Square Error “e037503°"" Observations (or Sum Wgts) 645 
Mean of Response 0.067764 Analysis of Variance 
Coservations (OL Sum gis) ie Source DF Sumof Squares Mean Square F Ratio 
Analysis of Variance Model 9 1.3972599 0.155251 131.1113 
Source DF Sumof Squares Mean Square F Ratio Error 635 0.7519141 0.001184 Prob>F 
Model if. 1.2489432 0.178420 126.2497 C. Total 644 2.1491740 <.0001 
Error 637 0.9002308 0.001413 Prob>F Parameter Estimates 
C. Total 644 2.1491740 <.0001 
Term Estimate Std Error tRatio Prob>|t| 
Parameter Estimates Intercept -0.06237 0.008775 -7.11 <.0001 
Term Estimate Std Error tRatio Prob>|t| LAE React InOrg SA 0.0117325 0.00233 5.04 <.0001 
Intercept -0.07178 0.009439 -7.60 <.0001 MAE Class Range 0.000004 6.888e-7 5.84 <.0001 
LAE React InOrg SA 0.011604 0.002545 4.56 <.0001 LAE Class Range 0.0000212 0.000001 19.46 <.0001 
MAE Class Range 0.0000039 7.521e-7 5.24 <,.0001 LAE Class Prob 0.0503286 0.006507 7.74 <.0001 
LAE Class Range 0.0000185 0.000001 16.12 <.0001 TCT Conceal -0.000549 0.000047 -11.78 <.0001 
LAE Class Prob 0.0492387 0.007107 6.93 <.0001 TCT Detect Range -0.003587 0.000451 -7.96 <.0001 
TCT Conceal -0.000533 0.000051 -10.47 <.0001 # LAE 0.0198858 0.000958 20.76 <.0001 
TCT Detect Range -0.003614 0.000492 -7.34 <.0001 (LAE Class Range-3569.77)*(LAE Class Range-3569.77) -6.907e-9 7.86e-10 -8.79 <.0001 
#LAE 0.0198858 0.001047 19.00 <.0001 (TCT Conceal-50.0155)*(TCT Conceal-50.0155) -0.000009 0.000002 -5.19 <.0001 
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Response Prop TCT Class 
Summary of Fit 


RSquare ATS Me, | 
RSquare Adj © 0.725647 2 
Root Mean Square Error **9.030259°"" 
Mean of Response 0.067764 
Observations (or Sum Wgts) 645 


Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 


Model 14 1.5723599 0.112311 122.6672 
Error 630 0.5768141 0.000916 Prob>F 
C. Total 644 2.1491740 <.0001 


Parameter Estimates 


Term Estimate StdGrror_ t Ratio Prob>|t| 
Intercept -0.062471 0.007717 -8.10 <.0001 
LAE React InOrg SA 0.0120452 0.002049 5.88 <.0001 
MAE Class Range 0.0000039 6.059e-7 6.50 <.0001 
LAE Class Range 0.0000212 9.586e-7 22.09 <.0001 
LAE Class Prob 0.0503922 0.005722 8.81 <.0001 
TCT Conceal -0.000549 0.000041 -13.40 <.0001 
TCT Detect Range -0.003615 0.000397 -9.11 <.0001 
#LAE 0.0198858 0.000842 23.60 <.0001 
(LAE React InOrg SA-0.00031)*(LAE Class Range-3569.77) 0.0000076 0.000002 4.38 <.0001 
(LAE Class Range-3569.77)*(# LAE-3) 0.000006 6.537e-7 9.19 <.0001 
(LAE Class Prob-0.65039)*(# LAE-3) 0.0137356 0.004044 3.40 0.0007 
(TCT Conceal-50.0155)*(TCT Detect Range-5.08527) -0.000093 0.000016 -5.85 <.0001 
(TCT Conceal-50.0155)*(# LAE-3) -0.000156 0.000029 -5.39 <.0001 
(LAE Class Range-3569.77)*(LAE Class Range-3569.77) -7.013e-9 7e-10 -10.02 <.0001 
(TCT Conceal-50.0155)*(TCT Conceal-50.0155) -0.000008 0.000002 -5.05 <.0001 


Note that the ratios used in the experimental design and their basic 
parameters (for example, ratio of MAE/LAE classification probabilities, MAE 
Classification probability, and LAE classification probability) are all included as 
potential terms in the model selection process. However, none of the ratios 
appeared in any of the stepwise models, while some of the basic parameters (for 


example, LAE classification probability) are included in the selected model. 


The adjusted R-squares as opposed to the standard R-squares for the 
three models are considered when selecting the appropriate base model for 
analysis so that the matrices are normalized against the number of terms used in 
the models. The full model (includes the quadratic and interaction terms) explains 
73% of the variability, while the main effect model and quadratic model explain 
only 58% and 65% respectively. The quadratic and interaction terms account for 
about 20% of the explanatory power in the full model. These terms are therefore 
deemed important and the full model is used a baseline model for analysis. 
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The actual by predicted plot on the left shows the lower tail of the 
distribution being cut off as the actual proportion can only take on values 
between zero and one. Similarly, there are no negative residuals when the 
predicted value is less than or equal zero, hence the residual by predicted plot 
appears capped by the 45 degree boundary on the bottom left of the plot. This 
departure from the regression assumption of additive constant variance noise 
would be of more concern if we were interested in making accurate numerical 
predictions of classification capabilities, rather than in identifying those factors 
that have the greatest impact on classification capability. It also shows the value 
of obtaining qualitatively similar results from several different analysis methods, 
such as partitioning trees, Multiple Linear Regression, and logistic regression. 





Actual by Predicted Plot Residual by Predicted Plot 
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The Pareto plot displays the factors of significance in descending order in 
accordance to their effects on the proportion of TCT classified. 


Pareto Plot of Transformed Estimates 





Orthog Estimate 
0.02812. 





LAE Class Range 

TCT Conceal 

(LAE Class Range-3569.77)*(LAE Class Range-3569.77) 
(LAE Class Range-3569.77)*(# LAE-3) 

TCT Detect Range 

LAE Class Prob 

(TCT Conceal-50.0155)*(TCT Detect Range-5.08527) 

LAE React InOrg SA 

MAE Class Range 

(TCT Conceal-50.0155)*(# LAE-3) 

(TCT Conceal-50.0155)*(TCT Conceal-50.0155) 

(LAE React InOrg SA-0.00031)*(LAE Class Range-3569.77) 
(LAE Class Prob-0.65039)*(# LAE-3) 0.0041515 
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The prediction profiler graph shows a significant increasing linear 
relationship between the response variable and the number of LAEs, which also 
appears in the Pareto plot as the factor with the greatest impact. The graph also 
illustrates the non-linear relationships of the response variable with both LAE 
classification range and TCT concealment ability. 


Prediction Profiler 
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0.00031 = 678413R Y 3569778 9S 0.65039 50.0155 = 5.08527 S 
LAE React MAE Class LAE Class LAE Class TCT Conceal TCT Detect #LAE 
InOrg SA Range Range Prob Range 


The interaction plot highlights the statistically significant interactions in 


solid lines. 


Interaction Profiles 
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The partitioning tree shows a partition of the mean proportion of TCT 
classification that explains approximately 68.5% of the response variability. The 
right branch of each partition represents the split that offers a higher mean 
classification rate than the left branch. The first few tiers of the tree depict the 


most significant factors. The results are consistent with those suggested by the 
earlier observations. 


































































































All Row: 
count sas R-square = 0.685 
an 0.0677643 
Std Dev 0.0577687 
# LAE<4 # LAE>=4 
Count 387 Count 258 
Mean 0.0446202 Mean 0.1024806 
Std Dev 0.0348481 Std Dev 0.0672464 
LAE Detect Range<4585, LAE Detect Range>=4585 LAE Class Range<2866 LAE Class Range>=2866 
Count Count Count 76 Count 
Mean 0.0293715 Mean 0.0596342 Mean 0.0378333 Mean 0.1294762, 
Std Dev 0.0197051 Std Dev 0.0397148 StdDev 0.0284235, StdDev 0.0599873, 

































































TCT Conceal>=88 TCT Conceal<88 TCT Conceal>=86 TCT Conceal<86 
Count 30 Count 165 Count 26 Count 156 

0.017267 an 00873374 Mean 0,0446923 Mean 0.1436068 

Std Dev 0.0149793 Std Dev 0.0379217 Std Dev 0.028548 Std Dev 0.0516129 


















































# LAE<2 # LAE>=2 LAE Class Range<3439 LAE Class Range>=3439 
Count 55] Count 110 Count 38 Count 118 

Mean 0,0391879|] Mean —0.0814121 Mean 0.1067719 Mean 0.554689 

Std Dev 0,0240523]| Std Dev 0.0357398 Std Dev 0.0332393 

















Std Dev _0.0509618 





The Multiple Linear Regression models are also provided for the 


proportion of armor classified with consideration of only the main effects (left), the 


main and quadratic effects (right) and including the two-way interactions (next 
page). 
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Response Prop Armor Class 


Response Prop Armor Class 


Summary of Fit 


Summary of Fit RSquare OMe 
RSquare Adj {0° 0773953 +, 
RSquare 0.546679 Root Mean Square Errors, 0.052919 _,* 
RSquare Adj eOBABOT-o, Mean of Response “prozac?” 
Root Mean Square Error an 0.07535 oe Observations (or Sum Wgts) 645 
Mean of Response 0.192601" 5 = 
. Analysis of Variance 
Observations (or Sum Wgts) 645 
Analysis of Variance Source DF Sumof Squares Mean Square F Ratio 
Model 9 6.1999396 0.688882 245.9965 
Source DF Sumof Squares Mean Square F Ratio Error 635 1.7782373 0.002800 Prob>F 
Model 7 4.3614987 0.623071 109.7406 C. Total 644 7.9781769 <.0001 
Error 637 3.6166781 0.005678 Prob>F Parameter Estimates 
C. Total 644 7.9781769 <.0001 
z Term Estimate 
Parameter Estimates Intercept -0.218018 
Term Estimate StdError tRatio Prob>|t| LAE React InOrg SA -0.014634 
Intercept -0.282286 0.028265 -9.99 <.0001 LAE React En Contact 0.0154657 
LAE React En Contact 0.0175678 0.005101 3.44 0.0006 LAE Airspeed 0.000569 
LAE Airspeed 0.000538 0.000126 4.28 <.0001 MAE Class Range 0.0000054 
MAE Detect Range 0.0000049 0.000001 3.42 0.0007 LAE Class Range 0.0000567 
LAE Detect Range 0.0000168 0.000005 3.61 0.0003 LAE Class Prob 0.0964538 
LAE Class Range 0.000032 0.000005 6.25 <.0001 #LAE 0.0310841 
LAE Class Prob 0.0926305 0.01425 6.50 <.0001 (LAE Class Range-3569.77)*(LAE Class Range-3569.77) -2.196e-8 
#LAE 0.0310841 0.002098 14.82 <.0001 (# LAE-3)*(# LAE-3) -0.022495 
Response Prop Armor Class 
Summary of Fit 
RSquare {0° 080845. 
RSquare Adj %, 0.804814 2: 
Root Mean Square Error “presorrs” 
Mean of Response 0.192601 
Observations (or Sum Wots) 645 
Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 
Model 12 6.4499617 0.537497 222.2841 
Error 632 1.5282152 0.002418 Prob>F 
C. Total 644 7.9781769 <.0001 
Parameter Estimates 
Term Estimate Std Error tRatio Prob>|t| 
Intercept -0.221601 0.018155 -12.21 <.0001 
LAE React InOrg SA -0.014474 0.00333 -4.35 <.0001 
LAE React En Contact 0.0158282 0.003331 4.75 <.0001 
LAE Airspeed 0.0005793 0.000082 7.06 <.0001 
MAE Class Range 0.0000056 9.857e-7 5.72 <.0001 
LAE Class Range 0.0000568 0.000002 36.48 <.0001 
MAE Class Prob 0.0617215 0.01893 3.26 0.0012 
LAE Class Prob 0.0626223 0.013932 449 <.0001 
#LAE 0.0310841 0.001369 22.70 <.0001 
(LAE Airspeed-170.109)*(# LAE-3) 0.0002518 0.000058 4.34 <.0001 
(LAE Class Range-3569.77)*(# LAE-3) 0.0000091 0.000001 8.58 <.0001 
(LAE Class Range-3569.77)*(LAE Class Range-3569.77) -2.191e-8 1.103e-9 -19.85 <.0001 
(# LAE-3)*(# LAE-3) -0.022495 0.001157 -19.44 <.0001 


Std Error 
0.019501 
0.003583 
0.003583 
0.000088 
0.000001 
0.000002 
0.010006 
0.001473 
1.187e-9 
0.001245 


t Ratio 
-11.18 
-4.08 
4.32 
6.45 
5.14 
33.85 
9.64 
21.10 
-18.49 
-18.06 


Prob>tt| 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 
<.0001 


As before, the full model explains 80% of the variability while the main 


effect model and quadratic model explain only 54% and 77% respectively. The 


interaction terms account for less than 4% of the explanatory power in the full 


model. These terms are therefore deemed insignificant and the quadratic model 


is the model of choice for further analysis. 
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The actual by predicted plot on the left shows the lower tail of the 
distribution being cut off as the actual proportion can only take on values 
between zero and one. In the same manner, there are no negative residuals 
when the predicted value is less than or equal to zero, hence the residual by 
predicted plot appears capped by the 45 degree boundary on the bottom left of 
the plot. These results are less pronounced that for the TCT classification model. 
There is also some evidence of nonlinearity, which may be due to the omission of 
interaction terms, or because the underlying relationships that reveal diminishing 


returns are not exactly quadratic. 























Actual by Predicted Plot Residual by Predicted Plot 
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RSq=0.78 RMSE=0.0529 Prop Armor Class Predicted 

















The Pareto plot displays the factors of significance in descending order in 


accordance to their effects on the proportion of armor targets classified. 


Pareto Plot of Transformed Estimates 





Term Orthog Estimate _ 
LAE Class Range 

#LAE 

(LAE Class Range-3569.77)*(LAE Class Range-3569.77) 
(# LAE-3)*(# LAE-3) 

LAE Class Prob 

LAE Airspeed 

LAE React En Contact 

LAE React InOrg SA 

MAE Class Range 
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The prediction profiler graph shows significant non-linear relationships of 
the response variable against the number of LAEs and LAE classification range, 
both of these factors and their quadratics appear in the Pareto plot as the highest 


four factors with the most impact on the proportion of armor targets classified. 


Prediction Profiler 


5 0.471 

S n 

2 0.274074 |= 
20 +0.0075 

a 





0.00762 





"0.00031 


; Oo = NX Q 9 rt cos wo 
0.00031 2 170.1098 = 6784.13N S$ 3569.77 8 S 0.65039 3 

LAE React LAE React LAE MAE Class LAE Class LAE Class #LAE 

InOrg SA En Contact Airspeed Range Range Prob 


The partitioning tree depicts a partition of the mean proportion of armor 
classification which explains approximately 83.6% of the data variability. The first 


few tiers of the tree capture the non-linear relationships of the classification rate 
against the number of LAEs and LAE classification range as well. 


All Rows 
Count 645 


R-square = 0.836 
Mean 0.1926014 
StdDev 0.1113035 
LAE Class Range<2579 
Count 























145 
0.0576078 
Std Dev 0.0325195 


Mean 











LAE Class Range>=2579 
Count 





500 
Mean —0.2317495 


Std Dev 0.094086 

























































































# LAE<3 # LAE>=3 
Count 200 Count 300 
Mean —_0.1487369 Mean —_0.2870913 
Std Dev 0.04971 Std Dev 0.0737993 
# LAE<2 # LAE>=2 # LAE>=5 # LAE<5 
Count 100]) Count 100 Count 100 Count 200 
Mean 0.1156525]| Mean —_0.1818212 Mean 0.236255 Mean —_0.3125094 
Std Dev 0.0297929]| Std Dev 0.0432232 Std Dev 0.0611649 Std Dev 0.061365 


























LAE Airspeed<166 LAE Airspeed>=166 LAE Class Range<3439 
Count 42 Count 58 Count 
Mean —_0.1945208. Mean —_0.2664763 Mean 
Std Dev 0.0459626 Std Dev 0.0525878 








LAE Class Range>=3439 
Count 
Mean 


62 











138 
0.3353841 
Std Dev 0.0581103 


0.2615948 
Std Dev 0.053543 
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3. UAV BEHAVIORS 


Contour plots showing the responsiveness of the LAEs toward newly 
classified enemy contacts and MAE redirections with respect to following the pre- 


assigned routings to support high target classification proportions. 








Contour Plot for Prop TCT Class Contour Plot for Prop Armor Class 
| Legend Legend 








Prop Armor Class 
— <=0.050 
_ <=0.150 
Ml <= 0.250 
Ml > 0.250 


Prop TCT Class 


<= 0.025 
_ <= 0.087 
MN <= 0.150 
MM > 0.150 














LAE React En Contact 
Oo 
1 
LAE React En Contact 
oO 
1 





























LAE React InOrg SA LAE React InOrg SA 








The dark blue patches show higher proportions of target classification than 
the light blue patches. It should be noted that the scales for the two contour plots 
are different. The plots show generally higher classification proportions on TCTs 
if an LAE is more reactive to MAE redirection, while no obvious benefit is seen by 
having an LAE be more reactive to a new classified enemy contact. For 
classifying armor targets, having LAEs react less to MAE redirection and more to 
a new Classified enemy contact seems to improve the overall classification 


performance. 
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4. IMPACT OF LAE FAILURE 


The following graphs depict the average proportions of TCT and armor 
classification due to the failure of an LAE at various time instances in the 


scenario with and without cooperative sensing capability. 
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